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ABSTRACT 



The National Workplace Literacy Program (NWLP) was a federal 
grant program that was initiated in 1988 to promote the following: 
contextualized, job-specific instruction in basic skills; joint governance 
through partnerships between employers, education providers, and employee 
organizations; and attention to individual workers' needs. The NWLP ' s impact 
was examined through a national evaluation with two primary objectives: 
describe the implementation and institutionalization of workplace literacy 
programs and assess the effects that workplace literacy instruction has on 
participating workers. The research focused on three local and two 
state-level partnerships funded in 1994. The study's major conclusions were 
as follows: program effectiveness is associated with instructional time; 
implementation is aided by experience and state/local infrastructure; and 
institutionalization is associated with workplace incentives for employers. 
Appended are the following: overviews of the structure and operation of the 
five program sites studied in depth; detailed profiles of the five sites, 
review of efforts to develop applied performance assessments; profile of 
impact study design and implementation at three local sites; data on workers 
in the impact study; estimation of program impacts and effects of hours of 
instruction on worker outcomes; and data collection forms. (Contains 32 
references and 19 tables/figures.) (MN) 
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EXECUTIVE SUMMARY 



INTRODUCTION 

The need to remain economically competitive in a changing global economy has led to 
widespread concern in recent years about the low literacy levels of many American workers. About 
40 percent of the nation’s workers, according to recent estimates, have limited abilities to perform 
basic reading, writing, and math tasks that are likely to be required by their jobs (Sum, forthcoming). 
Communication skills that include speaking ability also have become a principal requirement of 
successful employment in both the service and manufacturing sectors. The fact that increasing 
numbers of workers in the U.S. were bom abroad and have limited proficiency in English 
compounds the challenge of improving many workers’ literacy skills. 

One strategy for developing workers’ basic skills is known as workplace literacy. Rather than 
teaching literacy skills in the abstract, workplace literacy focuses on teaching literacy skills that 
workers need to do their jobs. A key aspect of workplace literacy is that the curriculum is based on 
job task analyses and instruction makes use of materials from the work site. 

With the goal of increasing the number of workplace literacy programs nationwide and 
developing models that could be followed by other workplace programs, in 1988 the U.S. 
Department of Education (ED) initiated a federal demonstration grant program known as the 
National Workplace Literacy Program (NWLP). The program was designed to emphasize (1) 
contextualized, job-specific instruction in basic skills; (2) joint governance through partnerships 
between employers, education providers, and employee organizations; and (3) attention to the needs 
of individual workers. It was anticipated that the program could benefit participating workers and, 
in turn, their employers. 

This is the final report of a national evaluation of NWLP partnerships, conducted by 
Mathematica Policy Research, Inc. (MPR) and its subcontractor. 1 Our in-depth study was guided 
by two major research objectives: (1) to describe the implementation and institutionalization of 
workplace literacy programs and (2) to assess the effects that workplace literacy instruction has on 
participating workers— and in so doing, demonstrate how rigorous research designs can be 
implemented to evaluate program impacts. We conducted our research in three local and two state- 
level partnerships funded in 1994, the last time NWLP grants were awarded. Brief descriptions of 
these diverse sites are provided in the accompanying box. 2 Additional details on our research 
methods are provided at the end of this executive summary. 



'COSMOS Corporation served as the subcontractor during the first three years of the study; in 
the fourth year Abt Associates became the subcontractor. 



2 Because of confidentiality agreements with the partnerships, in this report we do not identify 
the five sites, but rather refer to them by number only— the three local sites are referred to as Sites 
1, 2, and 3; the state-level programs are referred to as Sites 4 and 5. 
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THE FIVE IN-DEPTH STUDY SITES 



• Site 1. This program focused on providing English-as-a-Second-Language 
(ESL) instruction primarily to Asian immigrants, nearly all of whom were 
female, working in the garment industry in a large city. Classroom-based 
instruction was offered on weekends, at the offices of the main partner, an 
organization with a long history of teaching literacy skills and providing job 
training. The other partners were a garment workers union and an association 
of garment manufacturers; no employers were directly involved. The 1994 
grant was the partnership’s third consecutive one under the NWLP. 

• Site 2. This workplace literacy program was developed to serve employees of 
a single employer, a metal processing company located in a semi-rural area. 
Workers attended classes, held at the plant during working hours, that taught 
skills such as math, reading, writing, communication, problem-solving, and 
decision-making. A community college and a literacy organization were also 
involved in the partnership. The 1994 NWLP grant was this partnership’s first. 

• Site 3. The goal of this program was to improve the English language skills of 
Hispanic immigrants working in food processing and agriculture-linked 
businesses. During the grant period, the lead partner, a nonprofit employment 
and training organization, teamed with several companies spread over a fairly 
large geographic area. Classes were typically held at the work sites, 
immediately after the workday. The 1994 NWLP grant was this partnership’s 
first. 

• Site 4. This state-level program, led by the state community college board, 
involved the creation of local partnerships between an education provider 
(usually a local community college) and one or more companies. The common 
goal of the local projects was to improve literacy skills of workers in businesses 
that have adopted total quality management (TQM). Instruction focused on a 
wide range of skills, such as reading, writing, math, problem-solving, and ESL. 
The 1994 grant was the partnership’s third consecutive one under the NWLP. 

• Site 5. This site’s state-level partnership involved local partnerships between 
1 1 technical colleges and 20 manufacturing firms. Curriculum and instruction 
were tailored to the needs of employees at each participating company, with the 
program emphasizing self-paced, individualized learning through services and 
instructional materials at on-site learning centers. At the state level, the main 
partner was the state technical college board; other key players were a union, 
a university research center, and a statewide association of employers. The 
1994 NWLP grant was the fifth for this partnership. 



CONCLUSIONS FROM THIS STUDY 



Results from this study demonstrate that when appropriately implemented, workplace literacy 
programs can have short-term impacts on workers of a fairly broad scale. These outcomes range from 
literacy-related behavior at home to performance on the job. At the same time, it is important to note 
that most of the 1994 NWLP-funded workplace programs did not exhibit the features that appeared 
linked to the impacts observed in the in-depth study. The overall message from this study is that while 
it is possible to implement and institutionalize effective workplace literacy programs, doing so will 
present challenges to employers, program leaders, and policy makers. 

• Program Effectiveness Is Associated With Instructional Time. Our results suggest that 
instructional time is a potentially important element affecting overall program 
effectiveness. Substantially increasing instructional hours, however, will require major 
changes in the way most workplace literacy programs are constituted. Typically, workers 
in the NWLP programs amassed only modest amounts of instructional time. Structural 
forces in the workplace and on workers’ personal lives will need to be overcome to 
increase workers’ hours of instruction. Educating employers about the value of additional 
hours and actively coaching workers to enter appropriate follow-on courses in local 
colleges may help to increase instructional time. 

• Implementation Is Aided By Experience and State/Local Infrastructure. Initiating local 
workplace literacy programs is less demanding than keeping them functioning smoothly 
over time. A successful, on-going workplace literacy program requires a considerable 
investment of staff time, flexibility in adapting to a range of circumstances, training of 
quality instructors, and guidance from experienced professionals. Staff with years of 
experience found meeting these challenges less onerous. Where an infrastructure was in 
place through networks of experts affiliated with community colleges and with state 
agencies, local programs staff also found it easier to tackle these challenges. 

• Institutionalization Is Associated With Workplace Incentives for Employers. In the 
absence of public support, employers’ commitment of funds is critical in ensuring that 
local programs continue. Employer financial support was more evident in organizations 
that were restructuring workers’ jobs, and was less evident in work sites that 
accommodated workers’ skill deficits. Public or other private sources of funding may be 
necessary to sustain programs for workers whose job performance does not depend on 
mastery of many basic literacy skills. 



LESSONS ABOUT THE IMPLEMENTATION AND INSTITUTIONALIZATION OF 
WORKPLACE LITERACY FROM FIVE SITES 

Implementing Workplace Literacy 

At the local level (including the local partnerships formed under the two statewide programs), the 
experience of the in-depth study sites point to several common areas requiring vigilant attention when 
establishing and operating workplace literacy programs. 
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• Employer Participation and Support . Program staff typically needed to cultivate 
employer support and participation throughout the operation of a program. This involved 
a continuous process of refining goals and objectives for the work site, distinguishing 
workplace literacy instruction from job training and personal development, adapting to 
organizational and production changes in the workplace, and gaining support from 
supervisors and employees. 

• Time Necessary for Curriculum Development. Developing workplace curricula 
demanded a substantial investment of instructors’ time. Workplace literacy clearly is not 
a job limited to teaching time. The task of curriculum development was somewhat less 
time-intensive for instructors at sites with greater experience in workplace literacy. 
Instructors’ and employers’ perceived need for curriculum that suited individual 
workplaces made externally produced content of limited value. 

• Instructors’ Qualifications. Although most instructors had some teaching experience 
from other jobs, they still had to develop competencies specifically associated with 
workplace literacy, such as building organizational support and marketing the program. 
This was typically accomplished through on-the-job experience and mentor relationships, 
rather than through formal training. 

• Constraints on Instructional Time. The workplace typically imposed constraints on 
instructional time. For example, some employers wanted to hold down costs for paid 
release time and minimize interference with production schedules. Also, scheduling 
classes for too many weeks could reduce employees’ interest. Some sites tried to 
compensate for such limits by creating opportunities for participants to practice their skills 
outside of classes, through take-home exercises and audio cassettes. 

• Information on Program Effectiveness. Across sites, employers relied on trusted staff 
in the workplace and measures of program inputs to assess how useful their programs 
were. Workers’ performance on instructor-developed tests, attendance data, and anecdotal 
information were the main types of information used to assess programs’ effectiveness. 
Standardized literacy tests were infrequently employed tools for assessing how well 

. workers mastered skills taught in particular courses. 

Institutionalizing Workplace Literacy 

Because programs were started with federal funds, questions emerge about their ability to endure 
with other sources of support. The prospects for workplace literacy programs’ early continuation 
beyond the federal demonstration grants are promising. Some form of workplace literacy instruction 
was planned at most of the local partnership sites after the federal grants. Continuation depended 
primarily on employers’ commitment of their own funds. 

• Nearly two-thirds of the 37 local programs examined in this study were slated for 
continuation for at least one year. 
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• The 30 local workplace programs located in the two states with a statewide workplace 
literacy strategy in place experienced rates of institutionalization of 70 and 75 percent. 

This suggests that statewide infrastructure can be a factor important to continuation. 

• Employers whose workers reported less need for using English language skills to do jobs 
were less likely to continue ESL-oriented workplace literacy programs with their own 
funds (Sites 1 and 3). In comparison, employers seeking to gain international 
certifications of quality appeared more likely to commit company funds to continue the 
workplace literacy programs (Sites 2, 4, and 5). 

How long workplace literacy programs will continue after this initial support is unclear. 
Information from several continuation sites indicated that while these programs would continue, they 
were likely to change in response to employers’ and employees’ preferences, and the elimination of 
federal requirements. The changes involved programs becoming shorter in duration and including 
more job training and personal development components. 



The Role of State and Local Infrastructure 

For workplace literacy programs to endure and expand beyond the federal grants, the development 
of an infrastructure will be important. Infrastructure links interested employers (or employer 
associations or unions) with a readily available source of information about workplace literacy, not 
to mention the basic elements of such services-instructors and curriculum. It has two components: 
(1) state-level activities and leadership and (2) local providers of technical expertise and staff. Two 
state-level partnerships that sought to build a system of workplace literacy in their states two-year 
colleges (Sites 4 and 5) provided an in-depth picture of infrastructure issues. 

• Common Infrastructure Activities/Functions. Both sites carried out common activities, 
including (1) defining and disseminating a model of how workplace literacy programs 
should ideally be structured and operated; (2) providing professional development 
activities for instructors, such as through mentor relationships and effective-practices 
guidebooks; (3) cultivating networks of professional staff, such as through regular 
meetings and shared technical assistance; (4) generating data on program participation, 
and (5) helping to resolve problems and conflicts that might arise at the local level, such 
as the need to gain the support of front-line supervisors. 

• Core Process Versus Core Curriculum. Site 4 and Site 5 had somewhat different goals 
and strategies concerning infrastructure development. Site 5 focused on refining a 
framework of policies and procedures to expand workplace literacy programs throughout 
the state, specifying steps such as the analysis of literacy skills for various jobs and the use 
of steering committees. Site 4, in contrast, formed a coalition of experienced workplace 
instructors from a handful of colleges to develop a curricular model addressing the skills 
necessary to carry out TQM reforms in a variety of workplaces. Based on the experiences 
of the two sites, it appears that developing a core curriculum is more difficult than 
focusing on a standard process that allows different curricula to emerge. 
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• Keys to Success. Three features surfaced as important in state-led efforts to foster 
adoption of workplace literacy programs: (1) decentralization, (2) flexibility, and (3) 
support from local colleges. Decentralization is necessary to accommodate the traditional 
autonomy of community/technical colleges and to reinforce local adaptability. Flexibility 
is critical for responding to unique local situations, such as employer preferences. Support 
from colleges requires an organizational commitment of resources and leadership. 

Local Colleges and State-Level Organizations. Continuation of state-level infrastructure 
will depend largely on state funds, while continuation of local college infrastructure will 
be fueled by employers’ demands for assistance, public subsidies, or both. Site 4 had no 
ongoing sources of state support, whereas Site 5 maintained a state grants program similar 
to the NWLP. The presence of state support was linked to better prospects for the 
continuation of state and local infrastructure. 



PROGRAM IMPACTS ON WORKERS AT THE THREE LOCAL SITES 
Worker Outcomes 

Literacy instruction linked closely to the workplace can lead to a variety of positive short-term 
outcomes for participants. 3 It can change workers’ career and educational plans, alter their literacy 
habits at home, and improve their literacy skills and job performance. Table 1 displays all the program 
impacts we detected, by site. Most of the impacts were small or modest in magnitude, ranging from 
.2 to .4 standard deviations. 4 Importantly, while all three sites had some impacts, most impacts were 
heavily concentrated in one site. (The unique features of this site and the implications of this finding 
for other workplace literacy programs are discussed in the subsequent part of this overview.) 

• Changes in Workers ’ Plans and Skills. Workplace literacy can lead workers to change 
their career and educational plans, and can improve their literacy skills. For example, 
after NWLP courses ended, participants at Site 1 were more likely than nonparticipants 
to express an interest in taking other courses. In addition, participants rated their own 
literacy skills-such as reading and writing English-higher after courses ended than did 
nonparticipants. We also found rather substantial literacy skill improvements, at Site 1, 
as measured by a standardized literacy assessment. Finally, compared with 
nonparticipants, after courses ended participants in some sites reported engaging more 
frequently in certain literacy tasks at home, such as reading a variety of materials. 



3 Since employers were reluctant to keep applicants out of courses for longer than one or two 
course cycles, this study could not examine whether workplace literacy had any impacts on long- 
term outcomes. 

4 The following interpretation may make the magnitude of these program impacts easier to grasp: 
A difference of .2 standard deviations indicates that the average score for workers in the treatment 
group is higher than the scores of almost 60 percent of those in the control group; a .4 standard- 
deviation difference tells us that the average score among treatment-group members exceeds the 
scores for 66 percent of the control-group members. 
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TABLE 1 



OVERVIEW OF SIGNIFICANT PROGRAM IMPACTS, BY SITE 



Outcome ■ £ • &£ 


l§ siteiiiai 






Workers' self-rated ability to: 




/ 




read English 


/ 




understand English 


/ 


/ 




speak English 


/ 






write English 


/ 






use math 


/ 






solve problems/use reasoning 


/ 






Standardized literacy assessment 


/ 






Changed educational or career goals 


/ 




/ 


Planned to take: 


/ 






a computer course 






courses leading to a 2- or 4-year college degree 


/ 






a home-study course 


/ 






Signed up for another course 


/ 






How often workers read (at home): 








letters or bills 


/ 






coupons 


/ 






labels on food 


/ 






food recipes 


/ 


/ 




instructions 


/ 




street signs 


/ 






notes from a teacher or school 


/ 






TV Guide or other television listing 


/ 






How often workers write (at home): 




/ 




forms or applications 


/ 




appointments on a calender 




/ 




Workers' self-reported ability to work as part of a team 


/ 


/ 




Worker speaks English on the job 


/ 






Works as part of a team on the job 






/ 


Benefits received on the job: a 


/ 

/ 






paid sick leave 


b 

u 




paid holidays 


✓ 

✓ 


D 

w 




health insurance 


/ 


D 




Number of days missed work in last 12 months 






/ 


because of a job-related accident 


b 




Supervisor ratings of how well worker performs tasks: 


b 


/ 




arrive on time, ready for work 




/ 


follow supervisor's verbal instructions 




/ 


ask if coworkers or customers need help 






answer simple questions related to the job 




/ 


/ 


give clear directions/instructions to others 




/ 


complete a task on time 




/ 





NOTE: All impacts shown were significant at the .10 level using a one-tailed test. 

a Any short-term impacts on these outcomes are produced by a program’s direct emphasis on benefits during instruction or counseling. 
b Unable to measure or to estimate effect for this outcome at this site. 
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• Job-Related Outcomes. Workplace literacy can have significant impacts on workers’ 
experiences and performance on the job. After courses ended, participants at two sites 
experienced positive impacts on their teamwork skills and were also more likely than 
nonparticipants to report receiving various job benefits, such as paid holidays and health 
insurance. In addition, according to employers at those sites, participants missed work 
less often due to accidents and outperformed nonparticipants on several tasks, including 
giving clear directions to others. 



Factors Associated with Differences in Site Effectiveness 

Site 1 outdistanced the other two sites examined in the in-depth study in the number of impacts 
it had on workers (see Figure 1). More instructional hours and a combination of other features may 
be responsible for Site 1 ’s greater level of effectiveness. 



FIGURE 1 

SITES' PERCENTAGE OF OUTCOMES WITH 
SIGNIFICANT IMPACTS 
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• Importance of Instructional Hours . Two types of evidence suggest that instructional 
time may be a key to overall program effectiveness. First, sites where participating 
workers received more average hours of instruction had more program impacts across a 
broad range of outcomes. Site 1, where workers received an average of about 43 
instructional hours in the first course attended, had the greatest number of impacts; Site 
2, where workers received slightly less instruction (31 hours), had impacts on fewer 
outcomes; and Site 3, where workers received an average of just 19 hours of instruction, 
showed impacts on the fewest outcomes. Second, statistical analyses of data from all 
three sites showed that increases in the hours of instruction that individual workers 
received were associated with greater gains on 10 outcomes, including workers 5 self- 
reported ability to use math and how often they read various materials at home. 
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• Other Influential Factors. 

Differences between sites in the 
number and type of program 
impacts observed also may be 
related to numerous factors 
other than instructional hours, 
including program objectives 
and course scheduling. 
Although this evaluation could 
not statistically disentangle all 
these possible influences on 
program effectiveness, several 
practices distinguished Site 1 
from the other two sites and may 
be linked to this site’s greater 
effectiveness (see box). 



Promising Factors in Site l’s Effectiveness 

• Workers completing an average of 43 hours of 
instruction in initial course and 66 hours of instruction 
over an 18-month period. 

• Emphasis on employees’ advancement to jobs better 
than their current job. 

• Scheduling courses when employees were not fatigued 
from work and including child care plus reimbursement 
of transportation costs. 

• Following a highly structured curriculum that 
incorporated several modes of instruction and that was 
developed over a period of years by experienced staff. 

• Forging a close alignment between instruction and a 
standardized literacy test. 



The program features in Site 1 that distinguished that site suggest many differences from 
other workplace literacy grantees funded by the NWLP. These included a heavy emphasis 
on the advancement of the individual worker that was largely independent of employers’ 
immediate involvement. (This became a detriment for continuation of the program once 
public sources of support were no longer available.) The program in Site 1 also offered 
highly structured classes on weekend days when workers’ attention was less centered on 
the workday. Finally, Site 1 had developed its program over a number of years, resulting 
in a highly experienced staff and a very carefully structured mix of instructional 
experiences involving English use in the garment industry. 



STEPS FOR VALIDATING STUDY FINDINGS 
Research Framework for Extending Study Findings 

Several lessons emerge from this exploratory evaluation about how studies of workplace literacy 
might be designed and conducted. It is essential to define a framework to link future studies so they 
can have maximum use in validating and extending the findings in this study. This framework will 
permit others to examine program impacts across a variety of sites and by revealing the circumstances 
under which various program elements are most effective. 

• Evaluations should be based on random assignment and compare the outcomes of 
treatment and control groups to rule out other possible influences on worker outcomes; 
if the present study had simply compared participants’ skills before and after courses, it 
would have produced very different— and inaccurate— conclusions. 
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• Implementing validation studies will require public policy support and incentives to 
encourage sites’ participation. Employers who have already decided to participate in 
workplace literacy efforts are unlikely to invest in evaluations of the type required to 
extract generalizable findings; employers’ find it sufficient to rely on broad indicators like 
participation and anecdotal information from trusted sources within their own 
organizations to determine a program’s success. 

• Data collection should not rely heavily on local program staff who have other priorities 
to accomplish. 

• A nationally coordinated study of workplace literacy should use procedures, such as meta- 
analysis, that allow synthesizing results across different sites. The meta-analysis should 
strive to incorporate measures of a core set of outcomes, many of which were used in this 
study. Standardized literacy tests, however, should be used only in situations where they 
closely align with the curricula. 

• Identification of possible long-term effects produced by workplace literacy programs will 
require the longitudinal collection of information across a nationally representative sample 
of employed adults. This undertaking will require care in constructing items that yield 
data about adults’ participation in workplace literacy, the characteristics of that 
instruction, and other educational and employment aspects of their lives. 



STUDY METHODOLOGY 

To describe the implementation and institutionalization of workplace literacy programs, the 
evaluation team conducted case studies of a purposively-selected sample of five diverse NWLP 
projects-three local and two state-level partnerships. The case studies involved two visits at each 
partnership, conducted 12-18 months apart, to interview program staff, employers, and workers. 

To assess the effects that workplace literacy instruction has on participating workers, MPR 
implemented an experimental design at the three local sites and randomly assigned course applicants 
to either a treatment or control group. The treatment group was allowed to enroll in workplace literacy 
immediately; the control group had to wait at least until the next course cycle began. Data on many 
potential short-term outcomes were collected from workers, employers, and supervisors, typically both 
before classes started and after they ended. Program impacts were computed by comparing the 
average outcome for the treatment group with the average outcome for the control group. 

In addition to the rigorous impact analyses, we also explored the relationship between hours of 
instruction and worker outcomes. Specifically, we compared average hours of instruction received 
with sites’ overall effectiveness and also used an analytic model to isolate the effects of hours of 
instruction on participating workers’ outcomes. 

In conducting various analyses and describing the five in-depth study sites, we incorporated data 
from the National Workplace Literacy Information System (NWLIS), a computer database for 
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gathering data from NWLP partnerships, established as part of the overall national evaluation of all 
45 grantees funded by ED in 1 994. 5 



descriptive information on all 45 partnerships-including the partners involved, courses and 
instructors, and leamers-is provided in an interim report from the national evaluation; see Moore, 
Mary T., David E. Myers, and Tim Silva, “Addressing Literacy Needs at Work: A Profile of 
Institutions, Courses, and Workers in the National Workplace Literacy Partnerships.” A Report 
Prepared for the U.S. Department of Education, Planning and Evaluation Service. Washington, DC. 
Mathematica Policy Research, Inc., 1997. 
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I. INTRODUCTION 



Economic competitiveness and increased demographic diversity in the labor pool have prompted 
public concern about the low literacy levels of many U.S. workers. This concern has only intensified 
in recent years as historically low unemployment in certain areas has limited employers’ options to 
hire workers with desired skill levels. Instead, managers face the alternatives of developing the skills 
of workers already on the job, relocating to different labor markets, or recruiting employees from 
distant locations. 

Not all employers agree on how critical a problem skill deficits are in their own companies. 
Nevertheless, estimates from the National Adult Literacy Study (NALS) provide ample reason for 
concern. These estimates indicate that two-fifths of the nation’s workers demonstrate significant 
limitations when performing reading, math, and writing tasks that are likely to be part of their jobs 
(Sum, forthcoming). Such skill deficiencies are likely to place these adults at risk of job loss and 
diminished earning power in an increasingly technological, global economy. Moreover, workers’ 
low literacy levels pose a major challenge to economic growth since they hinder companies seeking 
to restructure production processes and assign front-line workers’ increased responsibility. Low 
literacy skills also exert a negative influence on workers’ lives outside their jobs. They can restrict 
parents’ ability to help their children educationally and individuals’ full participation in their 
communities. 

The skill deficits just described hinder many workers who grew up and attended school in the 
U.S. But the nation’s workforce now extends beyond these native-born workers to those who 
present other literacy challenges. During the past decade the nation has witnessed an increased 
number of immigrant workers entering the adult workforce— a development that complicates the 
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problem of literacy deficits in the workforce. Most immigrants take low-end, secondary jobs in the 
economy largely because over two-thirds have limited proficiency in English and many have fewer 
years of formal schooling than their U.S. counterparts (Rice and Stavrianos 1996). The marginal 
employment situations which many immigrants occupy threaten the stability of their families, 
hamper employers’ ability to realize the potential of this pool of workers, and slow efforts to fully 
integrate these adults into the community. 

The National Workplace Literacy Program (NWLP) which Congress established in 1988 
emerged as a way to develop workers’ literacy skills so they were better aligned with the new skill 
requirements of the changing economy. Administered by the U.S. Department of Education (ED), 
its purpose was to foster and spread exemplary programs to improve the work-relevant literacy skills 
of adults through instruction closely tied to the workplace. Between 1988 and 1994, ED 
competitively awarded NWLP grants to 345 partnerships. Each partnership had to be composed of 
at least one partner from a business, industry, or labor organization, and at least one partner from 
an educational organization, and had to be committed to delivering job-linked basic skills instruction 
to eligible adults. The NWLP defined basic skills broadly to include reading, math, English 
proficiency, communication, reasoning, problem-solving, and the completion of a high school degree 
or its equivalent. Eligible adults were individuals with inadequate skills who were 16 years or older 
and beyond the age of compulsory school attendance, and whose participation was likely to result 
in new employment, enhanced skills linked to continued employment, career advancement, or 
increased productivity. 
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This report is the final product of a national evaluation of the NWLP’s 1994 grants cycle in 
which 45 partnerships were funded for a period of three years. 1 The Planning and Evaluation Service 
within ED contracted with Mathematica Policy Research to undertake this study of the 
demonstration projects funded in this last cycle of workplace literacy grants. 2 In an earlier report, 
we provided a profile of the 1994 funded workplace literacy partnerships, that contains descriptive 
findings about who the partners are, what courses and instruction are offered, which workers 
participate, and what outcomes adult learners report upon completion of their participation (Moore, 
Myers, and Silva 1997). These findings confirmed the diversity of workplace literacy offerings and 
the limited amount of instructional time that the majority of workers spend in these programs, but 
suggested that the receipt of 30 or more hours of instruction may make a difference in employees’ 
reported job outcomes and ratings of their abilities. In this report, we report on a similar but more 
focused set of issues. We present results from an in-depth investigation of five partnerships 
purposively selected to examine the issues that affect successful implementation of workplace 
literacy programs and the continuation of programs once federal support has ended. Importantly, 
three of these partnerships provide evidence on the impacts of workplace literacy courses on 
workers. 



’Grants made between 1989 and 1994 covered a period of 1 8 months, while those made in 1988 
covered only 15 months. The extension of the grant period in 1994 allowed partnerships added time 
for start-up activities and carrying out demonstration and dissemination functions. 

2 The administration and Congress have not sought funds for NWLP grants beyond those 
awarded in 1994. 



A. BACKGROUND AND DESIGN OF THE NATIONAL WORKPLACE LITERACY 

PROGRAM 

Education in the workplace covers a large and diverse set of activities and participants. If we 
define education in the workplace as inclusive of formal job training, it is clear that a large fraction 
of employees and the vast majority of employers are involved. National estimates indicate that 
between 30 and 40 percent of adults engage in such training and between 70 and 80 percent of firms 
or establishments provide training opportunities (U.S. Department of Education 1997; Bassi, 
Gallagher, and Schroer 1996). This breadth of involvement, however, does not translate into broad- 
based participation within the workplace. The evidence is clear that hourly, front-line workers are 
least likely to have access to such training (Lillard and Tan 1992). 

A different definition of workplace education is useful for this report since formal job training 
encompasses instruction in a variety of skills other than basic education. Considering only that 
training which concentrates on basic academic skills, we find a dramatic drop in the percentage of 
firms and establishments that offer such programs to workers. Hollenbeck (1994) and Bassi (1992) 
have reported that between eight and ten percent of companies invested in this form of workplace 
education as of the early 1990s. Additionally, only about six percent of employees participated in 
reading, writing, and math training while at their current jobs, according to estimates from the 1991 
Current Population Survey (U.S. Department of Education 1997). Against this backdrop of 
generally low prevalence, there has been noticeable interest within certain states, community 
colleges, unions, and employers in increasing the availability of workplace education in basic skills 
to hourly workers. For the past ten years, this interest found a focus and source of support at the 
federal level though the NWLP. As the federal support ends, sustained interest and the 
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institutionalization of policies will be major factors in whether workplace literacy programs become 
more prevalent. 

The NWLP defines workplace education even more specifically than simply workplace- 
sponsored instruction with a focus on basic skills. Drawing upon concepts in the field of adult and 
secondary education that stress instructional connections with the contexts in which adults actually 
function, the NWLP incorporates several distinct emphases related to pedagogy, governance, and 
the anticipated beneficiaries of workplace literacy services. 



1. Contextualized, Job-Specific Basic Skills Instruction 

The NWLP approach departs from traditional forms of adult education and job training in its 
emphasis on closely tying curricular content to the basic skills that workers must possess to 
effectively perform specific jobs. Besides requiring job-oriented content, the NWLP emphasizes 
teaching methods that incorporate actual examples, problems, and materials from the workplace. 
The NWLP’s approach to workplace instruction grows out of research in the military and civilian 
sectors that shows basic skills instruction linked to job requirements is more successful than 
traditional techniques for teaching literacy skills to adults. 3 

The NWLP framework also distinguishes literacy skills that enable workers to competently 
perform a job from other job skills. Examples are helpful for illuminating these distinctions. For 
instance, knowledge of statistics and the performance of arithmetic calculations are basic literacy 
skills that employees must have to read blueprints or apply Statistical Process Control (SPC) 
techniques in the workplace. As such, the NWLP views them as within the scope of a grant. 



3 The research studies on functional context instruction based on workers’ jobs indicate that 
participants remained in programs at higher levels, retained their skills longer, and had improved 
more in performance of their jobs (Sticht 1975, 1987; Mikulecky and Lloyd 1993). 
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Instruction in reading blueprints and using SPC are considered job-specific skills rather than literacy 
skills, and, hence, fall outside the scope of a grant. While the intent of the NWLP is clear in terms 
of focusing on the more general basic competencies that workers must have to do certain jobs, where 
to draw the line is not as clear. ED encountered this problem when attempting to determine if 
computer skills fell within the group of basic educational skills covered by NWLP grants. 
Ultimately, ED decided that they did so only when linked to computer-assisted instruction, but 
educators and employers involved in workplace literacy have questioned the constraints that this 
determination has placed on their programs. 

The NWLP expects partnerships to complete a three-step process before they embark on 
actually instructing adult workers. Initially, to achieve a curriculum that integrates job-specific skills 
with literacy instruction, the NWLP workplace literacy partners must assess areas where workers’ 
educational skills may be inadequate. They then must analyze the specific literacy skills required 
in tasks workers perform in the course of their jobs. Next, the workplace literacy instructors proceed 
to develop a job-specific course of instruction that incorporates materials, problems, and exercises 
relevant to the work site. While this final step of developing curricula customized to job tasks and 
to the workplace is critical in the NWLP approach, the degree of customization and building from 
the ground up is left to each partnership’s determination. In the interests of minimizing 
inefficiencies from “reinvention of the wheel,” the NWLP governing requirements encourage 
partners to develop, reproduce, and disseminate curricula for transfer to similar businesses and 
industries. 
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2. Joint Governance Through Partnerships Between Education Organizations and 

Employers or Employee Organizations 

To strike an appropriate balance between job-specific and basic skills education, the NWLP 
relies on a governing partnership between organizations with expertise in each area. The partners 
establish a set of shared goals and objectives for the program of workplace literacy services, 
contribute at least 30 percent of the funding for services through cash or in-kind support, insure that 
the appropriate types of workplace literacy services are developed and implemented, and assume 
joint accountability for the federal grant. Education partners may be traditional institutions or 
nonprofit entities such as community-based organizations and manpower training agencies. 
Employers, labor organizations, and private industry councils are eligible to contribute the business 
and industry half of the partnership. Both state and local organizations can establish partnerships 
as long as they fit into the definitions of an eligible education or business partner, but all partnerships 
regardless of level must be engaged in the delivery of job-specific basic skills to adults. 

A long-term goal of the NWLP has been the institutionalization of workplace services. This 
institutionalizing of services was a key element included in the requirements governing the last cycle 
of NWLP funded partnerships. The requirements instructed partnerships to focus on the continuation 
and expansion of workplace literacy services after the end of federal funding, and the “integration of 
workplace literacy services into long-term planning of partner organizations” (CFR 472.22.h). This 
language suggests that service continuation is the primary institutionalization goal of the NWLP; the 
continuation of the partnerships as a vehicle for achieving this goal appears to be a decision left to 
partners’ preferences. 
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3. Attention to Individual Workers’ Needs 



The NWLP approach emphasizes attention to adult learners’ needs by arranging convenient 
access to the program and by taking into account individuals’ personal learning goals. Premised on 
current adult education concepts of best practice, the NWLP attempts to minimize the impediments 
to program participation— impediments such as transportation and child care during non-working 
hours. The location of workplace literacy courses is a key consideration in the NWLP framework 
with emphasis placed on learning environments that are readily accessible and conducive to adult 
learning. Settings within the workplace offer obvious advantages for meeting this criterion, but other 
locations also may be suitable since in some cases partner organizations may not have access to the 
workplace for such purposes. Educational counseling and individualized educational plans 
developed jointly with workers also are emphasized as critical services within NWLP partnerships. 

Importantly, the NWLP framework does not specify a number of features that various experts 
have debated as potentially contributing to a quality program of workplace education. The partners 
have broad latitude to determine the amount and level of instruction, set mandatory or voluntary 
requirements for workers, provide confidentiality protections to participating workers, establish 
selection and other participation rules for employees, and offer incentives such as fully or partially 
paid release time from work. 

4. NWLP’s Anticipated Beneficiaries: Employees and Employers 

A wide range of outcomes to benefit both employees and employers distinguishes the NWLP 
approach to workplace education. For employees, who may be employed or temporarily unemployed 
persons, participation in the workplace literacy program is expected to result in various outcomes: 
improved basic skills related to the job, increased readiness for training and promotion, a new job, 




s 29 



job retention, job advancement, and improved job performance. For employers, the NWLP 
encompasses potential benefits for two groups of employers: employers in general and employers 
who sponsor workplace literacy services. This distinction is important because the individual benefits 
resulting from workplace education are not always beneficial for an employee’s current employer. 
For example, new jobs and career advancement may mean employees left employers. 4 For employers 
in general, the NWLP has the potential to improve the pool of skilled workers in an area and, hence, 
may indirectly improve the productivity of employers who hire from this pool. For participating 
employers, a more direct set of benefits are expected. These include efficiencies from more effective 
workers and reductions in costs in a range of areas: declines in safety problems, employee errors, 
excess waste, employee turnover, and lost management time. 

Workplace literacy programs also may benefit employers in ways independent of any 
improvement in workers’ literacy skills. For example, employers may realize reduced tensions in 
labor negotiations, acquire quality certifications from national or international bodies, and find it 
easier to recruit job applicants. Although these outcomes may be important to employers and 
encourage their investment in workplace literacy programs, they are only important in the NWLP 
framework in so far as they are linked to services that also have improved the job-specific basic skills 
of adults, and hence employers’ productivity. 



4 The issue of employers’ reluctance to invest in workers’ portable skills, which include basic 
educational skills, has been debated in the research literature for some time. Some analysts have 
argued that employers’ reluctance is a major justification for public policy interventions to support 
workforce development. Others note that some employers contradict theory by supporting training 
that includes basic education, and, in fact, report that they realize returns on their investment in the 
form of increased loyalty from employees (Bassi 1994). 

O 

ERiC 



£0 



B. POLICY ISSUES CONCERNING WORKPLACE LITERACY 



Efforts to institute workplace literacy on a broader scale than what has existed in the past 
encounter several unresolved issues. Three major issues challenge the future of workplace literacy: 
(1) the feasibility of implementing and maintaining quality programs in the field, (2) evidence that 
workplace literacy programs are effective, and (3) the infrastructure necessary to successfully address 
basic literacy skills. among workers. 

1. Successful Program Implementation and Continuation 

An overview of services implemented across the NWLP partnerships is available from data that 
Mathematica collected from each partnership during the first year and a half of operation. 5 These data 
suggest that the programs organized by the partnerships generally conform to the approach 
emphasized in the NWLP requirements. Partnerships of education providers and business/industrial 
organizations oversee service delivery. Most individual partners are active in the areas of employee 
recruitment, establishment of operating procedures, provision of financial support, and monitoring 
of services. The courses have a job-orientation and are applied in nature. Moreover, most courses 
are located conveniently at the work sites and a majority are scheduled during or close to work hours. 
The instructors are well-educated and have opportunities for staff development focused on designing 
curriculum, assessing learners, and teaching in the workplace (Moore, Myers, and Silva 1997). 

This overview of the NWLP partnerships offers some insight into the workplace literacy services 
that have been provided through federal support, but it conveys only a general impression of 
implementation. The view is incomplete for informing questions about the impediments to smoothly 



5 These data were submitted by each partnership to Mathematica through the National Workplace 
Literacy Information System (NWLIS). They are reported in Moore, Myers, and Silva 1997. 
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functioning programs, and factors critical in the evolution and continuation of programs. To address 
these aspects of implementation, we investigate dimensions that illuminate the dynamic properties 
of programs. These dimensions include: 



• Employer Support and Commitment. Gaining employers’ on-going support in the 
context of the short time horizons common to the workplace is a key issue for workplace 
literacy programs. How vigorously employers participate in the planning phase and can 
set and communicate clear, widely shared expectations for the program are critical 
aspects of this issue. Proponents assert that support for workplace literacy is unlikely to 
take hold without nurturing, but nurturing efforts are not well understood. The ultimate 
tests of employer support and commitment occur when employers face continuation 
decisions and when workplace programs encounter changes in higher level management 
or economic reverses in the company. 

• Curricular Customization and Development. Several feasibility questions surround 
implementing job-specific curricula to increase workers’ basic skills. Do providers have 
sufficient time, resources, and expertise to identify employers’ needs with respect to 
basic skills and to analyze the job tasks for their skill requirements? Do efforts to 
customize content trivialize basic skills or teach skills with limited transfer to other jobs? 
Alternatively, do instructors fall back on traditional textbook exercises because they lack 
other resources? A final challenge for on-going programs is planning curricular 
sequences that connect discrete courses to enable workers to build upon and expand their 
skills. 

• Instructor Competencies and Employee Engagement. Teaching job-specific basic skills 
in a workplace setting places diverse responsibilities on instructors. Not only are 
instructors expected to design a curriculum linked to job requirements, they also are 
asked to recruit employees, serve as educational counselors, and promote workplace 
literacy courses to supervisors and management. The availability of qualified staff to 
assume these diverse roles and the capacity of programs to develop instructors’ 
competencies in multiple areas are crucial questions in implementing quality programs. 
Equally important is support for programs by employees. Many adult hourly workers 
have not been in formal learning situations for some time and may recall these situations 
being difficult for them. 

• Information to Support Decision-Making. The availability and adequacy of 
information are thought to be key factors in operating and maintaining quality programs. 
Instructors require information about changing job requirements, learners’ entry skills, 
learners’ progress, and the effectiveness of instructional approaches. Participating 
workers require information about course opportunities, competencies, and progress 
toward goals. Managers may need information about overall participation, met and 
unmet skill requirements in the organization, and whether expectations have been 
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achieved. Finally, external organizations that provide funding and technical support 
require information to assess programs’ effectiveness and areas where attention is 
needed. 

2. The Effectiveness of Workplace Literacy Services 

A major hindrance to expanding workplace literacy opportunities has been the limited evidence 
that is available about the actual effectiveness of these programs for employees or employers. Much 
of the available evidence is descriptive and based on observations from employers, employees, or 
program staff. Some researchers have conducted studies of specific programs using comparison 
groups of workers, but while their studies report some positive results, the lack of shared measures 
and potential selection bias between participating workers and their comparisons cast doubt on these 
findings (Mikulecky and Lloyd 1 993 ; Hargrove 1 989). Perhaps the most convincing evidence to date 
of the effectiveness of workplace literacy comes from a statistical analysis of data from two large 
nationally representative surveys: the Current Population Survey (CPS) and the National Household 
Education Study (NHES). This analysis indicates changes of 1 1 to 17 percent in increased earnings 
result from employees’ involvement in workplace literacy programs (Hollenbeck 1993). Even these 
findings, however, suffer from researchers’ inability to completely eliminate important motivational 
and other differences that may exist between workers who participate in workplace literacy and those 
who do not. 

Two issues have proven particularly challenging to obtaining evidence of whether workplace 
literacy programs result in impacts for learners and employers. First is arriving at a consensus about 
the appropriate outcomes and how best to measure them, particularly when these programs typically 
entail quite modest investments of learning time. Second is the difficulty of identifying results that 
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come from the services provided by a literacy program as opposed to a number of other occurrences 
in the workplace. 

a. Appropriateness of Behavioral Outcomes and Cognitive Literacy Outcomes 

The typical worker who participates in workplace literacy instruction receives a modest amount 
of instruction. 6 For example, workers who completed one or more courses in programs offered 
through the NWLP partnerships averaged only 30 hours of instruction, but half spent 16 or fewer 
hours in courses. These reports are consistent with those of other researchers who report few 
workplace literacy courses last more than 30 hours. Moreover, programs’ limited intensity as 
reflected in the hours of instruction for each worker is consistent with the time invested by hourly 
workers in workplace training in general. 

Given the briefness of instruction, cognitive measures such as standardized literacy tests have 
been perceived as inappropriate tools for measuring the skill improvements that adults may make as 
a consequence of participation in workplace literacy. It may be more appropriate to view these brief 
programs of instruction as producing an array of behavioral changes in adults linked to their literacy 
skills: personal insight into where skills are insufficient, more effective job performance, plans for 
future education, participation in further education, and performing more literacy relevant activities. 
Some researchers have underscored that workplace literacy courses, by definition, teach a narrow 
band of literacy skills— namely those found in specific tasks of a person’s job. Accordingly, they 



6 Although experts debate how much instructional time is typically necessary to produce a 
significant gain in cognitive literacy skills, between 50 and 100 hours are usually cited as necessary 
for producing gains of a grade level in reading (Moore and Stavrianos 1995). The median number 
of hours that the average adult receives from traditional adult education courses is 58 (Development 
Associates 1994). 
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argue that workplace literacy should be seen more as a gateway to improvement in general cognitive 
skills (Mikulecky and Lloyd 1993). 

In fact, few alternatives to standardized literacy tests are available for directly measuring changes 
in cognitive skills for participating workers that permit comparisons across similar workplace literacy 
programs. On one hand, customized tests prepared by instructors may offer some useful internal 
information to a program but they often are deficient as overall measures of skills taught and their 
application in the workplace. On the other hand, standardized literacy assessments have not been 
broadly embraced by workplace literacy instructors as appropriate measures of learners’ outcomes. 
Among the 45 NWLP partnerships, standardized literacy tests were used in only a third of all courses. 
(Use of such instruments is no doubt hampered by the logistics of administering pre- and post-tests 
that may consume an hour or more each within a course that only has 16 to 20 hours of instruction 
available.) The net result is a lack of any very effective instrument for assessing changes in cognitive 
skills for adults in low intensity workplace literacy programs. 

b. The Importance of Assessing the Value Added by Literacy Programs 

Workplace literacy services seldom occur independent of other changes in the workplace that 
also can influence workers’ productivity. Researchers have found that organizations’ adoption of new 
production systems— for example, those based on flexibility and teams— or new management 
structures involving work teams often is associated with the introduction of workplace literacy 
programs (Bassi 1994, Osterman and Batt 1993). Instruction linked to the workplace is frequently 
so embedded in other forms of restructuring that it becomes extremely difficult to decipher which 
outcomes, if any, are a function of the literacy training, and which would occur in its absence. For 
example, participating workers may rate their math skills higher after taking a mathematics course 
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linked to SPC, but similar workers not enrolled in courses may be experiencing the same sense of 
self-improvement simply by working with SPC on the job. 

The most widely recognized strategy for addressing such questions is use of a randomly-selected 
control group of employees from the workplace who have virtually identical personal characteristics 
and who perform the same jobs. The unique contribution of literacy services-that is, the value added 
by the literacy courses~can be identified by comparing outcomes for the two groups of workers since 
all workers will be subject to the same influences from changes in the workplace. Few studies of 
workplace literacy have used this method, however, for fairly obvious reasons. Individual workplace 
programs may have so few participant openings that the statistical comparisons of participants and 
nonparticipants are not feasible; recruiting the “right” number of participants is often perceived by 
instructors as more important than asking prospective candidates to wait for service openings; and 
employers may perceive the method as too restrictive and intrusive into workplace operations. While 
such practical impediments to uncovering solid evidence of workplace literacy’s presumed benefits 
may exist, the interests of public policy heighten the importance of overcoming these problems. 

3. Infrastructure for Providing Quality Workplace Literacy Services 

If evidence of impacts shows that workplace literacy services as defined by the NWLP improve 
employees’ basic skills and job performance, it does not address the question of how to systematically 
develop and sustain these services. Several issues are linked to development of an infrastructure for 
workplace literacy. Many employers, particularly those of medium to small size, have indicated that 
they perceive the cost and information necessary to launch (as opposed to operate) a workplace 
program within their companies as major obstacles (Hollenbeck 1993). Inefficiencies resulting from 
many stand-alone projects engaged in developing pedagogical techniques and curricula with which 
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other practitioners have experience, in theory, can be reduced by on-going arrangements for 
coordination, communication, professional development, and dissemination. Moreover, a well 
articulated system that links participating workers to additional educational opportunities beyond the 
workplace program would appear to be a key requirement if the short-duration literacy programs in 
the workplace are to start workers toward longer term opportunities that yield improvements in a 
broader range of cognitive skills (Grubb 1996). 

The above issues suggest the value of a durable infrastructure that connects the independent 
efforts of individual employers who invest temporarily or for some sustained period in workplace 
literacy programs. Some NWLP grants since the program’s inception have sought to foster statewide 
support structures for programs of workplace literacy. Among the 45 partnerships funded by ED in 
1994, seven focused on these state-level activities. Furthermore, several states have invested in 
infrastructure with earlier support from the NWLP or from their own sources. 7 An obvious platform 
for building statewide infrastructure has been the system of two-year community or technical colleges 
within many states. Other states that lack a well-developed system of two-year public colleges have 
chosen new or existing state agencies to build capacity among loosely-linked, independent providers 
of education and training services for adults. 8 The success of these efforts in achieving an on-going 
system of support, as distinct from simply initiating workplace projects in the state, is an important 



7 We are unaware of any comprehensive inventory of statewide workplace literacy initiatives that 
is available at this time. Among the state-level partnerships funded by the NWLP in 1994 were 
Alabama, Colorado, Maryland, Massachusetts, New York, Washington, and Wisconsin. In addition, 
Osterman and Batt (1993) describe four states’ efforts (California, Illinois, North and South 
Carolina) to develop broader employer-centered education and training activities. 



Illinois, for example, funds some local workplace literacy efforts in the state through the 
Secretary of State’s Literacy Office. A Workplace Literacy Training Institute has been established 
to improve the quality and effectiveness of local workplace programs (Burke and Ellingson 1997). 
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question for the future of workplace literacy efforts. A key aspect of this question is identifying the 
critical functions for an enduring statewide or regional infrastructure, and exploring promising ways 
of carrying them out. 

C. GOALS AND DESIGN OF THE IN-DEPTH EVALUATION 

The results described in this report are based on research designed to advance current knowledge 
about workplace literacy’s implementation, institutionalization, and effectiveness. Rather than 
prematurely undertaking a large scale, nationally representative study of impacts and implementation 
which might be impractical to apply in the field, this study looks in depth at five diverse workplace 
literacy partnerships and tests the application of rigorous evaluation tools in three of these 
partnerships. Thus, besides attempting to responsibly inform the central issues surrounding 
workplace literacy as described earlier in this chapter, this evaluation aims at constructing a model 
to guide expanded efforts to determine the effectiveness of workplace literacy programs. 9 

1. Reliance on a Purposive Sample of Partnerships 

In identifying the partnerships for the in-depth study, we selected a set that showed reasonable 
progress toward putting in place several features that various researchers and practitioners have 
observed are important to successful workplace literacy programs (see box) (Alamprese 1994; Kutner, 
Sherman, and Webb 1991). This selection strategy increased the likelihood that a qualified program 
of workplace literacy was in effect to produce outcomes. We also sought diverse partnerships that 
would bracket a range of contexts and clienteles, and reflect the local employer or union focus as well 



9 The NWLP required all partnerships to arrange for an external evaluator to conduct formative 
and summative evaluations of each partnership. These reports are independent of the national 
evaluation and are submitted by the partnership to ED at the conclusion of each three-year grant. 
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as the state-level focus. The latter was important 
to understanding the development of appropriate 
infrastructure for workplace literacy services. 

Other critical factors in choosing local 

partnerships were the potential for producing 

0 

sufficient levels of enrollment and partners’ 
willingness to accommodate random selection of 
a control group of workers who would delay 
their entry into the workplace literacy program 
for several months. 10 The final requirement for 
selection into the sample was willingness to use 
a standardized literacy test appropriate to a 
program’s curriculum. 

The five partnerships comprising the final sample for the in-depth evaluation included three local 
partnerships and two state-level partnerships. 11 Intentionally, we only applied random selection in 



Features Experts Have Linked to Effective 
Workplace Literacy Programs 

• Partners active in activities, focused on explicit 
goals, and supported by top management 

• Staff with expertise in teaching adults and 
developing job-relevant curriculum; opportunities 
for staff training 

• Needs assessment and job skills analysis 

• Operational plan linking activities and objectives 

• Curricular materials related to jobs and using 
workplace materials 

• Sufficient instructional time and opportunities for 
practice 

• Incentives for workers to participate 

• Strands of linked or alternative courses available 

• Support services such as education counseling, 
child care, and transportation 

• Learners assessed and given periodic feedback 



,0 We discuss the design requirements in greater depth in Chapter III. Generally, the statistical 
requirements for each candidate site called for approximately 100 participants and 100 controls. 
These numbers could accumulate over the course of the evaluation. A sample of this size detects 
a difference of one-third of a standard deviation between treatments and controls on a specific 
outcome of interest. Larger sample sizes would detect smaller differences between treatments and 
controls. 

11 We conducted phone interviews with all partnerships to assess their general progress in 
implementing features important to a workplace literacy program. Based on these interviews, we 
reduced those under consideration to 27. Next, we made in-person visits to 14 partnerships to obtain 
further information and to ascertain support for an impacts design at the sites. This stage resulted 
in a prioritized list of 10, of which five partnerships attended a training session and three fully 
implemented the evaluation. Overall, approximately 942 employees were included as respondents 

(continued...) 
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the local partnerships selected for the study. The next chapter describes in much greater detail the 
key characteristics and implementation experience of each sampled partnership. The major obstacles 
to obtaining a larger number of impact sites were reluctance to use a standardized literacy test and 
employers’ unwillingness to entertain the random selection of workers from shifts and teams. 
Although inadequate enrollment levels at first appeared to be an obstacle, our review of enrollments 
reported by the partnerships across 1 8 months suggests that over two-thirds would have enrolled 
sufficient numbers at the end of three years to accommodate this type of design. 

2. Impact Assessment Focused on Short-Term, Diverse Employee Outcomes 

Our investigation of the effectiveness of workplace literacy services emphasizes detecting a range 
of behavioral and cognitive changes after employees complete one course of enrollment in workplace 
literacy (see Figure 1.1 where shaded areas designate the concepts covered in this evaluation). This 
emphasis is justified by the typically short duration of instruction and the fact that across all 
partnerships only a quarter of participants enroll in more than one workplace literacy course. 
Furthermore, it is extremely difficult to expect changes in workplace productivity or returns on 
investment to result from a program of literacy services if participating employees fail to show 
changes on an array of varied outcomes. Moreover, if as a first step in evaluating the impacts Of these 
programs we can uncover evidence of impacts on workers after the early phases of involvement, we 
would suspect later effects might also emerge. Stated differently, we saw a lack of early impacts as 



"(...continued) 

in the impacts study. Resources were a consideration in the number of partnerships that could be 
part of the final study sample. 
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lowering the odds that outcomes such as increased wages and promotions would occur much longer 
after instruction. 12 

Equally critical to focusing the evaluation of effectiveness on measures of individual employee 
change is the inclusion of a wide range of behavioral measures. Employees’ future education and 
training plans, employment outcomes such as applying for new jobs or experiencing greater 
responsibility in a current job, performance as assessed by supervisors, self-assessments of skills in 
core areas of literacy, and literacy behaviors at home and in the community all are included in this 
examination of programs’ impacts. Standardized literacy test scores also constitute an important 
component of the research framework’s efforts to inform the debate about the appropriateness of 
using these cognitive measures to assess the outcomes of workplace literacy programs. 

3. Qualitative Approach to Implementation, Institutionalization, and Infrastructure Issues 

The challenges confronting implementation of quality programs of workplace literacy, the 
factors influencing decisions to continue providing these services, and employers’ perceptions of 
programs’ contribution to productivity are topics most effectively understood through on-site 
observations and interviews. To understand the evolution of implementation and institutionalization, 
the evaluation team made two site visits to each of the five partnerships, to review documents, to 
interview key actors including instructors and employers, and to conduct focus groups of workers. 
Visits were separated by 12 to 18 months, with the last visit occurring close to the end of the three- 



12 Long-term impacts that occur in the absence of observable changes in the short-term are 
possible, as evaluations of early childhood interventions such as the Perry Preschool Program in 
Michigan have shown. These preschool interventions were sustained over several years, however, 
making them demonstrably different from short duration efforts such as workplace literacy. Even 
so, literacy skills did not appear to improve for the Perry pre-school students once they reached 19 
years of age (Sticht 1993). We also emphasize that a failure to detect short-term impacts may be the 
consequence of insufficiently sensitive measurement tools. 
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year NWLP grant. While the evaluation sought to find quantitative measures of organizational 
productivity linked to the workplace program, equally important was identifying the evidence that 
employers relied upon to assess the effectiveness of literacy services. 

D. ORGANIZATION OF THE REPORT AND TERMINOLOGY 

Three chapters form the remainder of this report. Chapter II describes in detail the five 
partnerships included in our in-depth examination of workplace literacy. The chapter presents 
findings about the issues associated with successful implementation of programs, the 
institutionalization of programs by employers or partnerships, and the development of statewide 
infrastructure to support workplace literacy services. The third chapter summarizes workplace 
literacy programs’ impacts on participating workers, based on data gathered in the three local sites 
in which an experimental evaluation design was applied. Chapter IV discusses the conclusions that 
can be drawn from the study and presents a recommended framework for validating the findings 
about the effectiveness of workplace literacy in a larger number of sites. 

Throughout this report, we refer to the five partnerships as numbered sites (that is, Site 1 through 
Site 5). This is both a convenient shorthand as well as a way to preserve confidentiality for individual 
employers and program staff who agreed to participate in the national evaluation on this condition. 
Readers may wish throughout the report to recall that Sites 1, 2 and 3 are locally oriented partnerships 
that are independent of any state organizing structure. These partnerships were the focus of the 
impacts study. Sites 4 and 5 are state-level partnerships which were not included in the impacts 
study. 
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II. LESSONS IN IMPLEMENTING AND INSTITUTIONALIZING 
WORKPLACE LITERACY 



Three general observations emerge within this chapter. First, as we expected from our criteria 
for selection of in-depth sites, all five sites in the study were able to establish and maintain credible 
programs of job-customized literacy services for workers over the course of the three years of NWLP 
support. Second, regardless of major differences in their approach and clientele, all five encountered 
similar implementation issues. Some of these issues emerged at specific points in the three years 
for which the partnerships were federally funded, while other issues were continuously present. The 
sites addressed these challenges with varying degrees of success. Not surprisingly, the partnerships 
with previous experience in developing workplace literacy services, or that could tap into a network 
of support services, generally found the navigation of implementation issues somewhat easier. 

The third observation that emerges is that workplace literacy programs often continued after 
federal NWLP funding, but continuation varied among the sites. For local workplace programs, 
continuation usually depended on employers who had become persuaded of the value of workplace 
education programs and were willing to pay for them for at least one additional year. For state 
partners and education providers, the central question was the continuation of infrastructure, that is, 
the capacity and expertise to develop and guide workplace literacy programs. The continuation of 
infrastructure was less certain than the continuation of local programs and often depended on access 
to a combination of unpredictable funding sources. These included, for example, a steady market 
of interested employers willing to contract with education providers, and state funds or tax credits 
to stimulate employer investment and support staff in education provider organizations. 

To investigate implementation and institutionalization of workplace literacy, we studied the 
experiences of five quite different NWLP-funded partnerships. Two levels— state and local— were 



important in this inquiry. Because the NWLP required that all partnerships demonstrate the delivery 
of quality workplace literacy services to workers, each of the five sites we sampled contributed to 
understanding the issues tied to local service delivery. Therefore, when considering local service 
delivery questions, we included the experiences of all five sites in our analysis. In fact, while we 
conducted case studies in only five sites, we were able to consider the implementation experiences 
of 37 local programs. For the patterns we report about statewide implementation and 
institutionalization, however, we have restricted our observations to the experiences of the two sites 
that were state-level partnerships. 

In the next two sections of this chapter we describe salient aspects of each sampled site to reveal 
how the sites differ from each other and from the national profile of partnerships receiving 1994 
grants. Subsequently, we summarize lessons about how the five addressed issues involved in 
designing and delivering local workplace literacy programs. In the final section of the chapter, we 
compare the two state-level partnerships’ efforts to develop and sustain workplace literacy 
infrastructure. 

A. PROFILES OF THE FIVE PARTNERSHIPS IN THE IN-DEPTH STUDY 

A brief description of the five partnerships is important for orienting readers to each site’s 
unique context and approach to workplace literacy. The profiles below underscore how varied the 
five sites were in their focus, context, and scale of operations. A more detailed summary of each 
site’s program is included in Appendix A. 

• Site 1: Industry-Specific ESL at a Literacy and Job Training Provider. This urban- 
based partnership provides ESL instruction linked to the garment manufacturing 
industry. The courses are provided by the main partner, a nonprofit organization that 
provides literacy and job training. The nearly 600 workers who participated in the 1994 
NWLP grant were primarily immigrants from Asia, many of whom had been referred 
to the program by a union, the second of the three partners. Word-of-mouth also was 



a source of recruits. A less active third partner, an association representing industry 
employers, collaborated in designing the curriculum. The program focuses on 
improving English and communication skills so workers can obtain better jobs in the 
industry. Courses are voluntary and held on weekends at the premises of the main 
partner. Workers’ current employers have little or no involvement in the workplace 
program. Taught by experienced, bilingual ESL instructors, the curriculum is highly 
structured and has been honed over three successive NWLP grants by the project 
director and a full-time curriculum specialist. Video and audio tapes, computerized 
instruction, and volunteer tutors supplement the instructors’ presentations and class 
exercises. The union now includes some pieces of the ESL curriculum in its educational 
programs. The main partner, however, has not found replacement funds to continue 
offering workplace literacy services. 

Site 2: Employer-Embedded Literacy Instruction Initiated with a Local College. This 
partnership is centered at a primary-metal processing plant in a semi-rural area. 
Restructuring of production and the plant-wide adoption of statistical monitoring and 
control processes led the employer to enlist the local community college’s assistance in 
starting a work-based education program to improve the math, reading, writing, 
communication, problem-solving, and decision-making skills of the workforce. 
Obtaining international quality certification (ISO 9002) also was a major motivation. 
Located at the plant, the program has served over 300 of the plant’s 600 workers 
through this its first and only NWLP grant. Two local literacy organizations have been 
ancillary partners, providing tutors and some basic skills refresher courses. As the 
official grantee, the college initially helped staff the program but has been less active as 
the employer and on-site project director have assumed responsibility. Workers 
participate in classes on either a mandatory or voluntary basis; a learning lab and library 
are available on site for workers to use on their own time. Classes occur during the 
workday and accommodate shift and day workers’ schedules. The employer has hired 
the workplace literacy staff as company employees to continue the program next year. 
The courses will change to four-hour workshops to improve workers’ attendance and to 
minimize shift conflicts. 

Site 3: ESL at Diverse Employers Through an Employment and Training 

Organization. Serving about 500 workers across a rural geographic area, this new 
NWLP-funded partnership has focused on upgrading the job-relevant ESL skills of 
immigrant and native-born workers in food processing and agriculture-linked 
businesses. Although diverse, participating employers share goals of improving safety 
practices on the job and reducing high turnover of workers. The partnership is led by 
a nonprofit employment and training organization which formed partnerships with five 
employers to offer two levels of ESL instruction and GED preparation for employees 
at their workplaces. Instructors with ESL experience are teamed with aides to teach 
classes before and after work hours at each workplace. Workers’ participation is 
voluntary and uncompensated. The quality of available classroom space varies across 
employers. Instructors typically combine cooperative learning techniques with 
traditional ESL pedagogies for teaching listening and speaking: repetition drills, 
dialogues, and dictation. Materials from each workplace supplement a curriculum that 



includes a core list of ESL workplace competencies and competencies specific to 
individual work settings. Mandatory overtime, demands at home, and transportation 
schedules have hindered employees’ class attendance. Three employers have 
discontinued their workplace literacy programs and the remaining two have yet to 
decide. 

• Site 4: College/Employer Coalition for Teaching Total Quality Management (TQM) 
Literacy Skills. This state-level partnership combines experienced workplace literacy 
instructors from four community colleges and one community-based literacy 
organization under the leadership of a state community college board. The state board’s 
project leader and the instructors have developed local workplace programs with ten 
employers from the manufacturing, hospitality, health care, and financial services 
sectors. The team’s expertise and contacts with employers were fostered through two 
previous NWLP state-level grants. The third grant has focused on improving the 
literacy skills of ESL and non-ESL workers in organizations that have adopted TQM. 
The TQM approach requires skills in team building, problem-solving, decision making, 
and communication. The grant has aimed at developing a core curriculum template for 
local instructors to use in customizing curricula to different workplaces and building 
infrastructure to support workplace programs in the state. Incentives, schedules, and 
content vary considerably across employers. Instruction also varies, including a mix of 
formal classes, short classes, team instruction, and individualized tutoring. Skills taught 
include reading technical materials, writing, applied math, ESL, making presentations, 
solving work-related problems, and learning strategies. Seven employers have 
continued a work-based education program in some form subsequent to NWLP funding. 

• Site 5: Learning Centers Jointly Operated by Local Community Colleges and 
Employers. This state-level partnership has operated since 1988 with funding from five 
NWLP grants and annual funds from the state technical college board. Its recent NWLP 
grant focuses on expanding statewide infrastructure and starting new employer-based 
learning centers. The recent federal grant has served over 3,500 workers. Eleven 
technical colleges formed partnerships with 20 local manufacturing employers to 
develop employer-based learning centers with NWLP funds. The state-level partnership 
is led by the state board and includes the local partnerships and three other state 
organizations: a union, a university research center, and a state employers’ association. 
The state-level partnership’s workplace literacy framework emphasizes individualized 
instruction, self-paced learning, and voluntary participation, but local partnerships have 
flexibility in applying these elements. The state requires employers to cost-share in cash 
and to assume a greater fraction of the learning center’s cost each year. An experienced 
workplace-literacy coordinator in each college assists on-site instructors in conducting 
job literacy analyses and building programs of customized instruction. Workshops, 
computer-based instruction, and tutorials are common instructional formats. A wide 
range of skills involving math, writing, and communication are taught. Fifteen 
employers from the recent grant have continued a learning center on their own. 
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B. THE FIVE SITES COMPARED WITH NATIONAL ESTIMATES AND EACH OTHER 



A major intention in selection of the sites was bracketing a wide range of workplaces and 
approaches to providing workplace literacy services. Overall, the wide net cast was highly effective 
in capturing the intended diversity, as the profiles above suggest. Additional perspective on this 
diversity can be gained from using the information obtained from all NWLP partnerships from the 
1994 cycle to compare the five in-depth sites with a nationwide picture of participation and service 
delivery, and using the same information to compare the sites with each other. 

All NWLP partnerships reported data about various dimensions of their programs through the 
National Workplace Literacy Information System (NWLIS), a computerized information system 
constructed for this study. These dimensions included participating workers’ background 
characteristics, features of the courses they took, the amount of instruction they received, and 
characteristics of their jobs. 1 The data reflected the first 18 months of sites’ operations. Tables 
II. 1 -II. 3 display the results of these reports for all partnerships nationwide (referred to in the first 
column as national results) and for each of the in-depth sites. These tables highlight several aspects 
that distinguish the sampled partnerships from the national group as well as from each other. 2 



'Participating workers also were asked to report their own assessments of their skills at the point 
they began courses and at the point of completion, educational plans, and key events that had 
occurred related to their jobs (for example, better shift assignments, added responsibility, and 
awards). These data are included in the analysis of impacts reported in the next chapter and are not 
reported here. A detailed comparison of the NWLIS results for the five in-depth partnerships is 
included in Appendix B. National estimates from the NWLIS are described in a previous report 
from the national evaluation (Moore, Myers, and Silva 1997). 

2 Readers should be aware that the NWLIS data in Tables II. 1 -II. 3 combine results for an entire 
partnership. Consequently, the estimates for Sites 3, 4, and 5 reflect participation across all local 
employer locations in the program during the first half of the NWLP grant. The estimates for Sites 
1 and 2 reflect the single location where each provided instructional services. 
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DEMOGRAPHIC CHARACTERISTICS OF PARTICIPATING WORKERS 



lllllficipahts* BackgiH>uili(ll;lllllllMjl 
Characteristics 


National 






In-Depth Sites 




Hill 


Bill 


3 


. ' 4 




Age (average in years) 


38 


39 


41 


34 


37 


39 


Percent female 


50 


92 


7 


44 


65 


41 


Race/Ethnicity (percent) 


White 


55 


0 


45 


11 


56 


92 


Black 


17 


0 


50 


0 


7 


4 


Hispanic 


18 


0 


3 


88 


3 


2 


Asian/Pacific Islander 


8 


100 


1 


<1 


34 


1 


American Indian/Alaskan Native 


2 


0 


1 


1 


1 


1 


Foreign born (percent) 


. 25 


100 


3 


45 


45 


2 


Limited English proficient (percent) 


30 


99 


23 


79 


37 


9 


Education (percent) 


8 years or fewer 


11 


27 


2 


54 


9 


1 


9- 1 1 years 


18 


30 


10 


18 


13 


9 


1 2 years or more 


67 


14 


88 


18 


67 


89 


Undeterminable 


4 


30 


1 


10 


11 


1 
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TABLE II.2 



PARTICIPANTS’ EMPLOYMENT STATUS AND WORK CHARACTERISTICS 



' Participants’ E mploymeni|:$^^ 
Work Characteristics 


National 

Results 






iii|Depth Sites !§f§§ 




|| ill! 


2 


i 3 


1 4 




Employment Characteristics (percent) 


Not employed 


3 


13 


0 


2 


0 


<1 


Work more than 1 job 


19 


<1 


35 


9 


24 


21 


Member of a union 


23 


61 


1 


41 


16 


39 


Job Benefits (percent) 


Receive paid health insurance 


90 


53 


97 


80 


96 


96 


Skills Reported Necessary for Job (percent) 


Read instructions 


86 


37 


90 


44 


89 


93 


Receive instructions spoken in English 


89 


37 


92 


68 


96 


92 


Speak English 


93 


39 


98 


60 


100 


98 


Write English 


83 


13 


93 


38 


87 


96 


Use math 


78 


33 


90 


40 


74 


93 


Work as part of a team 


93 


59 


98 


69 


100 


94 


Solve problems/use reasoning 


87 


48 


92 


52 


96 


96 
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TABLE II.3 



CHARACTERISTICS OF WORKPLACE LITERACY PROGRAMS 





UNationaL f 
|"Resni^^; 






Iii-Depth Sites 




0du|^|?:Enrbiiirneitt 


nia 


2 


i-l|: |!| 


¥111: 


5 || 


Percent Taking Only One Course 


75 


74 


80 


62 


86 


NA 


Hours of Instruction Participants Received 


Mean hours 


30 


66 


40 


27 


22 


NA 


Median hours 


16 


63 


36 


19 


19 


NA 


Percent Enrolled in Courses Scheduled 


During workday (not lunch) 


71 


0 


100 


45 


51 


56 


Before/after workday 


44 


0 


16 


100 


37 


55 


At lunch 


5 


0 


13 


3 


0 


33 


On weekends 


3 


100 


13 


0 


0 


0 


Other 


7 


0 


8 


0 


20 


23 


Percent of Employers/Unions Providing 
Support or Incentives 


Provided or paid for transportation 


4 


100 


0 


0 


0 


7 


Provided or paid for child care 


1 


100 


0 


0 


0 


7 


Partial paid release time 


31 


0 


0 


0 


14 


74 


Complete paid release time 


28 


0 


100 


0 


86 


0 


Award certificate at completion 


53 


0 


100 


67 


86 


58 


Award ceremony at completion 


36 


0 


0 


67 


14 


36 

i 


Cash bonus at completion 


6 


0 


0 


0 


0 


7 



NA: Not available due to site’s reporting conventions. 
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The national picture of NWLP workplace literacy participants shows a diverse population of 
middle-aged workers, both native and foreign-bom. As a group, these workers were enrolled for 
modest amounts of time in a wide variety of courses at their places of employment. Workers usually 
took only one workplace literacy course which was held during the workday or at either end of it. 
The large majority of workers had jobs that provided paid health insurance and required employees 
to apply a broad array of basic literacy skills. Less than a quarter of participating employees 
belonged to a union and even fewer reported having a second job besides their current job. 

Each of the five in-depth sites departs from this composite view in one or more ways. With the 
exception of participants’ average age and the percent enrolled in a single course, the sites differ on 
a number of basic dimensions. Site 1 , for example, was heavily dominated by female workers, while 
Site 2’s enrollment was nearly all male. Site 1 focused on workers from Asia or the Pacific Islands. 
Sites 2 and 5, however, had only negligible numbers of participants bom outside the U.S. 

The five in-depth sites reflect two fairly distinct themes that tie together some of their specific 
differences. Some of the variability among the five sites is linked to sites’ tendency to focus on one 
of two groups of workers who have been major targets of the NWLP: workers with limited English 
proficiency and workers in restructuring organizations. It is clear that Sites 1 and 3 heavily focus 
their attention on ESL workers; in contrast, Sites 2, 4, and 5 heavily focus on workers in 
reorganizing workplaces. While the latter sites include participants with ESL needs, their workplace 
needs are broader than ESL. These underlying differences in the target population highlight the 
following key contrasts among sites: 

• Sites 1 and 3 both address different but fairly homogenous ethnic populations. Workers 
in both sites have considerably fewer years of formal schooling than participants in 
other partnerships. A number hold jobs that do not provide health insurance and 
demand less in terms of basic skills. In fact, a significant fraction of workers in both 
sites hold jobs that do not require reading instructions in English, writing in English, or 



even speaking English. In Sites 1 and 3, instruction is generally off-the-clock and 
scheduled for times other than the workday. 

• In contrast, the sex, racial/ethnic backgrounds, and union membership of workers in 
Sites 2, 4, and 5 differ considerably from the national profile and from each other. The 
sites also are very different in the jobs held by participating workers in terms of paid 
health insurance and need for a broad array of basic skills. A large fraction of 
employers in Sites 2, 4, and 5 provide paid release time for course participation, 
although this does not necessarily mean that instruction at each site always occurs 
during the workday. 



It is important not to overstate the extent to which differences cluster around sites’ targeting of 
a certain group of workers, however. Even after these differences are taken into account, significant 
variety still characterizes individual sites. For example, Site 1 holds its instructional sessions on 
weekends and includes support for transportation and child care. Site 3, however, due to workers’ 
geographic distance from their jobs, provides instruction during the week primarily before or after 
the workday is over. Furthermore, Site 1 participants, on average, accumulated 66 hours of 
instruction in the first 18-months of the workplace program compared with only 27 hours 
accumulated by participants in Site 3. 

The five in-depth sites bracket an additional core difference among partnerships: the extent to 
which employers play a central role. Site 1 differs demonstrably in this respect from the other sites. 
Site 1 adopted an industry-wide view to develop its workplace literacy curriculum; its program has 
limited connections with current employers of participants. Courses are held at the education 
provider, away from workers’ jobs which are spread throughout the city. This arrangement is 
depicted in Figure II. 1 alongside the other four sites’ organization of local workplace literacy 
partners and course locations. 3 As can be seen, the other four sites tied participants’ employers much 
closer to the partnership and to courses than did Site 1 . 



ERIC 



3 The dotted lines in Figure II. 1 represent limited involvement of a partner. 
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ORGANIZATION OF LOCAL NWLP PARTNERSHIPS IN THE IN-DEPTH STUDY 
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Site 2 is nearly the polar opposite of Site 1 . Site 2 is a highly employer-driven workplace 
literacy program in which over time the community college and other partners have receded into the 
background. A few similar situations in which employers played the dominant role were observed 
in Sites 4 and 5, but the more typical practice in other sites was for education providers and the 
employers to share direction of the workplace education program throughout the period of NWLP 
support. 

Importantly, while the organizational approach in Site 1 is not frequently used by NWLP 
partnerships, it offers this study an interesting model that may be particularly useful for extremely 
small businesses. Such businesses are unlikely to have sufficient funds to support stand-alone 
education programs for their employees or to have space available for instructional use. Courses 
physically separated from the workplace offer a solution to these problems, as does the development 
of a common curriculum for an industry. Similarly, employer-driven approaches to workplace 
literacy have relevance in this study because they constitute a major option for programs that are 
fully supported by employers. 

C. LESSONS ABOUT LOCAL IMPLEMENTATION AND INSTITUTIONALIZATION 

OF WORKPLACE LITERACY SERVICES 

1. Implementation Issues Encountered by Sites 

Below we discuss observations about local implementation that emerged from the five in-depth 
sites. Although there were instances where the sites approached a common issue similarly, often 
their approaches diverged. In the following discussion, we highlight the sites’ most striking 
similarities and differences. Generally speaking, Site 1 differed the most in how it addressed the 
challenges of implementation. Site l’s differences frequently resulted from offering workplace 



literacy instruction separate from employees’ workplaces and having quite limited contact with 
participants’ current employers. 

a. For Most Sites, Maintaining Employer Participation and Support Was Critical 

Throughout the Operation of the Workplace Program 

Obviously obtaining employers’ involvement and support was essential at the beginning of most 
workplace literacy programs, but sustaining these inputs also was essential and often proved quite 
challenging to sites. Sites had to resolve a variety of questions linked to employers’ on-going 
involvement and support. Importantly, as programs of instruction in the workplace became 
operational, the responsibility for addressing these questions typically fell upon instructors with the 
senior on-site instructor becoming the workplace literacy program’s recognized spokesperson. 
Assuming this responsibility became a source of difficulty in sites where instructors were 
inexperienced, isolated, and not expecting to play such a role. 

The off- work-site program at Site 1 also faced similar issues of how to maintain inputs from 
partner organizations, but in this case it was a union’s support in referring and counseling workers 
and input from an employers’ association to improve curricula to reflect changes in the clothing 
industry. Problems associated with instructors becoming spokespersons for workplace literacy were 
irrelevant in Site 1, however, due to the centralized location of courses. In fact, instructors at Site 
1 were explicitly limited to teaching roles. 

Maintaining employers’ involvement typically entailed: 



• Refining Goals and Objectives for the Work Site. Employers embarked on programs 
for a variety of broadly stated reasons— for example, to stabilize a workforce through 
reduced turnover or to acquire an international certificate of quality. Agreeing on how 
literacy instruction specifically would contribute to the broad goal required a careful 
development of specific expectations. In the early phases, senior managers typically 
worked with the more senior members of the educational partner’s workplace literacy 
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team to begin to address these matters, but instructors often became the central actors 
in this process once programs were running. 

• Distinguishing Workplace Literacy Instruction from Job Training and Personal 
Development. Questions in this area were often precipitated by how federal program 
guidelines interpreted basic literacy skills. Strong interest arose at various times in most 
sites to expand workplace literacy instruction, either to the personal realm (for example, 
teaching conversational English instead of job-specific English in Site 1 and teaching 
U.S. history in Site 3) or to the technical realm (for example, teaching the TQM skills 
in Site 4 or a plastics injection procedure in Site 5). Offering computer instruction arose 
as an issue in most sites. While raising debatable issues, these questions were important 
to keeping the job-specific and basic literacy focus of a workplace literacy program. 
Occasionally, disagreement about such matters led employers to withdraw from the 
NWLP partnership, but they often maintained an education program. 

Defining workplace literacy also entailed deciding how programs should relate to other 
forms of training within a company. The Human Resources division was the usual 
organizational “home” of workplace literacy in the in-depth sites, but this placement did 
not necessarily settle turf issues. An instructor at a Site 5 employer responded 
creatively to these issues by drafting a plan for management that spelled out the mission 
and links among training functions spread throughout the company. 

• Adapting to Changes in the Workplace and Gaining Support from Supervisors and 
Employees. Short time horizons in the workplace required workplace literacy staff to 
stay abreast of important changes in production and management. To learn about 
scheduling or technical innovations, instructors needed frequent contact with front-line 
supervisors and employees. The potential for staff turnover in management, however, 
necessitated developing the awareness of managers at different levels in the 
organization. Broad understanding of the program offset the problem of the workplace 
literacy program becoming the “baby” of a single manager, whose departure could 
jeopardize the program’s continuation. Task forces and committees of employees were 
a common technique that sites used to address these requirements. It took effort to get 
employees and managers to assume these roles. Nevertheless, opportunities to circulate 
on the floor and speak informally with supervisors were considered invaluable for 
gaining insight into production schedules and anticipated changes. 



b. Developing Workplace Curricula Demanded a Substantial Investment of Instructors’ 
Time 

Developing curricula customized to employees’ jobs was intrinsically a time-intensive process 
in the five sites. Having experienced instructors and curricular models to draw upon was helpful in 



O 

ERIC 



36 



59 



somewhat reducing the time needed to develop courses. As a general rule, novice sites struggled 
with the development of appropriate curricula more than the veteran sites. 

The bootstrap experiences of Sites 2 and 3, both first-time NWLP grantees, illustrate the effort 
required to developing curricula from the ground up in new workplace settings with new staff. The 
staff in both sites had to teach themselves how to conduct job task analyses to determine relevant 
literacy skills involved and then implement these analyses. Next, they had to transform literacy 
skills into work-related competencies and formulate work-relevant exercises for teaching these skills. 
Refinements and the substitution of new exercises were necessary as new groups of workers enrolled 
in the courses or participating workers moved into a subsequent course. Site 2 staff repeated this 
multi-step process in four divisions of the plant, producing five courses over the three years. Site 
3 used the process to construct basic and intermediate ESL courses to address specific competencies 
in five workplaces. 

Even when programs were staffed by experienced instructors and had access to outside 
expertise, only some of the time-consuming aspects of curriculum development disappeared. 
Experienced instructors at times were able to convert previously constructed curriculum into new 
courses and often knew where to find relevant published materials, but task analyses of jobs still 
were required when working with new departments or employers, and instructional formats 
demanded frequent revision and updating. For example, although veterans in the field of workplace 
literacy, the instructors in Site 4 felt strongly that they had to devise the curriculum at each 
workplace from the ground up. In these instructors’ experience, the availability of a project- 
supported team of curriculum developers and working with veteran teachers from other work sites 
did not remove the need to inductively develop curricula for each site. Yet, despite general rejection 
of a top-down approach, staff in several sites found the idea of inventories of workplace literacy 
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curricula appealing, and some remained optimistic that core curricula could streamline applications 
at new work sites. 

c. Instructors Developed Expertise in Workplace Literacy Primarily Through Experience 

and Mentor Relationships, Not Through Formal Training 

Sites seldom reported difficulty finding instructors for their workplace literacy programs, 
largely because instructors’ competencies were developed after they were hired. One respondent 
summarized this approach as finding creative, out-going individuals and fostering their skills in 
work-based education. Instructors came to the workplace literacy programs from diverse 
backgrounds, but most had some form of teaching experience either in ESL, K-12, or adult 
education. Even taking into account instructors who started with a background in teaching adults 
occupation-related skills, very few instructors had actually taught in the workplace before becoming 
involved with workplace literacy programs. Once hired as instructors, they found it necessary to 
develop a range of competencies important to maintaining a smoothly functioning program- 
competencies such as building support within the organization and marketing the program to 
employees and supervisors. Furthermore, echoing the time demands of curriculum development, 
most instructors in the in-depth sites had to spend time acquiring an understanding of the technical 
and industrial processes that defined jobs in specific industries or plants. 

In the sites that had limited prior experience with workplace literacy (Sites 2 and 3), instructors 
were largely self-taught with some assistance from either a curriculum specialist or the program’s 
evaluator. In contrast, the veteran NWLP sites relied on somewhat more formal approaches using 
mentor instructors from the colleges or, in the case of Site 1 , a full-time curriculum specialist who 
closely supervised the instructors’ classroom practices. Mentors and support networks of instructors 
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were particularly helpful in overcoming instructors’ isolation at worksites that could support only 
one or two part-time instructors. 

d. Instructors Routinely Faced Constraints on Available Instructional Time. 

The experience of the in-depth sites reinforces observations that workplace literacy programs 
are an abridged form of literacy instruction. For sites basing instruction at the workplace, instruction 
ranged from short workshops of around 8 hours to courses that lasted nearly 50 hours. 4 Only in Site 
1, where courses were separated from the workplace and held on weekends, did a single course’s 
instructional hours exceed 50. As noted earlier, enrollment in a sequence of courses was not a very 
effective path for accumulating additional hours since less than a third of employees did so in any 
of the sites. 

Instructors in workplace-based programs grappled with a number of factors that limited the time 
workers actually spent in instruction: 

• Employers were sensitive to holding down costs for paid release time and to minimizing 
interference with production schedules. 

• Workplace pressures encouraged the broad distribution of instructional opportunities 
instead of building the total hours of a single group of workers. 

• Employees were subject to production cycles that required a full complement of workers 
during regular shifts and considerable amounts of overtime. 

• Attendance lagged when sessions added to the workweek (for example, Site 3 sought 
six hours each week) and when courses lasted a number of weeks (for example, Site 2 
concluded 13 weeks was too long to retain shift workers in courses). 



4 The following were typical of the scheduled hours for a single course in the sites: 54 hours in 
Site 1; 40 hours in Site 2; 48 hours in Site 3; 8 to 30 hours for the local programs in Site 3; 8 to 20 
hours for Site 5, although Site 5's emphasis on individualized instruction made estimates difficult 
to obtain. 
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. Attendance was reduced when employees faced lengthy commutes to their jobs and 
family demands such as childcare. 

Staff in several sites took steps to compensate for these limitations. One route was to expand 
the time that participating workers had for practicing their skills. In some programs this meant 
arranging for tutors for participants outside the regular work schedule. In Site 4 tutoring also took 
the form of a “buddy” program for workers in ESL courses. This approach teamed English and non- 
English speaking employees to practice conversation skills on breaks or after work. A Site 4 
hospitality employer also called upon supervisors to reinforce the speaking of English on the job. 
Audio- and videotapes that could be taken home also were available at several sites. Tapes produced 
by Site 1 staff were aired through a local cable station that served immigrant communities. 
Traditional homework, however, usually was not very effective for extending the practice of skills 
taught in courses. 

A second route was to offset hindrances to course attendance by having supervisors and other 
workers actively promote courses and by providing assistance for childcare and transportation. 
Support for childcare was particularly important in Site 1 where workers could bring their children 
along to the weekend classes. In other sites, the large majority of employers did not provide 
childcare support, reasoning that courses held at the workplace made it unnecessary. However, some 
workers indicated they did not enroll in classes scheduled outside work hours because they had to 
supervise children at home. Whether financial assistance would have led them to arrange for a 
caregiver at home or in a center is uncertain. Childcare as an impediment to course participation, 
however, does appear to be complicated by a number of logistical and personal concerns. 
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e. Instructor-Designed Tests, Attendance Data, and Anecdotal Evidence of Improved 

Productivity Were the Main Types of Information Used in Sites 

Feedback to instructors about improvement in workers’ skills and the effectiveness of courses 
is important for deciding next steps for individuals and for the instructional program as a whole. 
With the exception of Sites 1 and 3, instructors in the in-depth sites based these decisions on tests 
they had designed for each course. These tests varied considerably; some were constructed to reflect 
job-specific competencies, such as filling out forms or simulations of team problem-solving, but as 
a general rule, the tests reflected material taught in the course as opposed to competencies that were 
applied in workers’ jobs. Employee surveys, course evaluations, and informal feedback from 
supervisors also were often used to assess the effectiveness of the instructional program. 

Not surprisingly, staff generally considered standardized literacy assessments unsuitable for 
measuring the types of skills taught in specific courses. s This view prevailed much less in Sites 1 
and 3, the ESL-focused sites, where there was greater reliance on standardized assessments (for 
example, assessments of oral language proficiency, a state writing assessment, and a state basic skills 
assessment). In Site 1 especially, perhaps due to the longer time it had spent in developing its ESL 
curriculum and refining its pedagogical philosophy, the skills stressed by the standardized oral 
proficiency assessment fit closely with skills emphasized in the basic ESL course. Such close 
alignment between a standardized test and instruction was rare in other sites. 

Few local workplace literacy staff in the five sites had sufficient time and expertise to develop 
assessments that measured literacy skills in the form of competencies applied on the job. To 
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5 Site 2 and 3 had agreed to administer a standardized literacy test, the Comprehensive Adult 
Student Assessment System (CASAS), for the national evaluation’s purposes, but had not planned 
to rely on this test in all their courses. Site 3 used the Adult Language Assessment Scale (LAS) 
throughout the grant period for most workers. The LAS test focused on oral English proficiency 
but did not cover other basic skills. 
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improve upon available measures of workers’ gains in the realm of cognitive literacy skills, staff 

from the national evaluation team collaborated with Sites 1, 2, and 3 — the sites in the impacts study 

to institute applied performance assessments of workers’ literacy skills. 6 Site 1 had already adopted 
a performance assessment of garment assembly and used this opportunity to develop two additional 
applied assessments that measured learners’ competence in reading and interpreting charts and 
tables. The two other sites had made less progress in transforming skill topics into actual 
competencies, which is an important precondition to developing performance measures. After 
making this translation, the sites produced applied performance assessments that they agreed to use 
in placing participants and gauging their progress at the end of courses. Overall, the development 
of applied performance assessments proved quite challenging for the sites. It demonstrated that 
individual project development of technically sound performance assessments was not advisable as 
a future solution to the deficiencies of standardized literacy tests. Local staff generally do not 
possess the technical background and training that is critical to the development of reliable and valid 
assessments. Outside expertise offers a more promising approach for constructing appropriate 
assessment tools. 

Employers decisions about workplace literacy programs seldom were based on evidence of 
workers gains on literacy tests. Employers turned to other sources for decisions about the 
program’s effectiveness-sources that typically were less exacting than business’ image of hewing 



6 A review of the experience and lessons learned from this undertaking are included in Appendix 
C. Many of the problems that sites faced concerned constructing pre- and post-assessments of 
equivalent levels, scoring rubrics for oral and written items, and including items that sampled the 
full range of competencies addressed by the curriculum. 
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to a hard bottom-line suggests. 7 Three sources of information commonly contributed to managers’ 
assessments of program effectiveness: 

• counts of course enrollment and completion 

• anecdotes about critical incidents associated with the workplace literacy program (see 
box) 

• statements summarizing barometers of company or unit performance (for example, 
employee turnover, safety incidents, down time, meeting of production targets, rework 
orders, and job training time). 



The first of these sources satisfied employers that the instructional programs could recruit and retain 
a sufficient number of workers. The last two were used to address whether the programs made a 
difference in organizational performance. Anecdotes 
about critical incidents were by far the most common 
source of evidence about the program’s contribution to 
performance. Even when barometers of company 
performance such as reduced employee turnover or 
safety incidents were cited as measures of the workplace 
literacy programs’ effectiveness, employers rarely 
produced calculations or tabulations to support the 
citations. Instead, summary statements that the 
barometers had changed (for example, less spent on 
training) sufficed. Whether internal budget documents 
contained analytic breakdowns of relevant measures was 



Sample Anecdotes of Critical Incidents 

(Site 5) 

"Prior to the program, employees had 
trouble accurately completing job tickets in 
the machine shop. Now, those problems do 
not exist.” 

“An employee averted downtime on the 
production line thanks to the training that he 
received through the workplace education 
program. Had the employee not interceded, 
the cost of the downtime would have 
exceeded the annual operating budget for 
the education center.” 

"Every employee in the east plant was 
required to complete a 4.5-hour, hands-on 
course in measurement and gauging. As a 
result of the training, measuring accuracy 
improved company-wide and a reduction in 
rework produced documented savings.” 



’Employers in Site 1 were not consumers of information about the effectiveness of the 
workplace literacy program because of their limited involvement and awareness of it. 
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unclear. We concluded that if more empirical measures existed, they were extremely closely held 
by management and difficult to obtain. 

Employers and instructors in the sites were of two minds regarding the information available 
about programs’ effects on organizational performance. On one hand, they were not troubled by 
having to infer causal connections. Managers readily acknowledged the workplace literacy program 
was one of many possible influences on companies’ or units’ performance. The observations of 
supervisors and other anecdotes were compelling to these managers because they emanated directly 
from known individuals who were close to actual production and services. On the other hand, 
employers and program staff in some sites conceded that more information related to quantifiable 
measures such as turnover and safety infractions would have been helpful at the point the federal 
grants were ending and questions of continuation were before them. 



2. Institutionalization of Local Workplace Literacy Programs 

Significantly, most local workplace literacy programs continued after the NWLP grant. Nearly 
two-thirds of the local workplace literacy programs in the five in-depth sites continued in some form 
after the federal NWLP assistance stopped, just over a quarter discontinued, and the remaining 
fraction had not been decided at the point of the last site visit (Table II.4). Programs followed two 
principal paths of continuation. On one path the employer absorbed the function by hiring relevant 
staff. On the other, the employer continued to collaborate with an education provider. We regarded 
either approach as continuation of a program of services. However, we did not regard the use of a 
site’s curricular materials by other entities as constituting continuation of a program of instruction. 
Some materials prepared by Site 1, for example, were included in courses offered by its partner 
union, but we did not classify this as continuation of the site’s services. 
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CONTINUATION OF LOCAL WORKPLACE LITERACY 
PROGRAMS AFTER NWLP SUPPORT 3 
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37 


14 
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“Based on decision to continue at least one additional year. 

b Number continued with education provider plus number absorbed by employer, divided by total number of local 
programs. 



a. Factors Influencing the Continuation of Workplace Literacy 

The encouraging record of continuations came primarily from the three sites that focused on 
workers in restructuring organizations: Sites 2, 4, and 5. The sites that primarily served a population 
of ESL-workers, Sites 1 and 3, were less successful in institutionalizing their workplace literacy 
programs. Our strong speculation is that these differences are associated with the workplace 
accommodations to the ESL population that participants in these two sites reported. As we noted 
earlier in this chapter, workers in the ESL-focused sites indicated their jobs required fewer skills than 
in the other sites, particularly skills in reading, writing, and speaking English. It is reasonable to 
infer that employers in these sites experienced limited pressure to absorb the cost of ESL-oriented 
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literacy programs once external subsidies were removed. 8 In contrast, several of the employers in 
continuation sites experienced opposite pressures. Jobs in these sites demanded a full array of skills 
from workers, and management in many of these sites had embarked on a quest for international 
certifications of quality that necessitated repeated demonstrations of literacy skills. 

The factors mentioned above suggest one important reason behind whether programs continued 
or discontinued. A host of organizational changes such as the departure of key personnel, new 
ownership, and competing priorities also influenced a program’s fate. All of these reasons were 
advanced as explanations for the discontinuation of workplace literacy programs. They reinforce 
the importance of program staff maintaining employer support at several levels throughout the 
operation of the workplace literacy program. 

Site 5’s experience with local program institutionalization is particularly noteworthy in 
suggesting a potential payoff from financial policies that accustom employers to paying for 
programs. The site’s policy of graduated cost-sharing required participating employers in the third 
year of the program to supply at least 75 percent of the cost in cash. This policy appeared to have 
accustomed employers to finding room in their own budgets for workplace literacy and weeded out 
employers who did not have any long term expectation of supporting literacy programs on their own. 

b. Continuation Issues for the Long Term 

Program institutionalization as reported here, however, must be subjected to two cautions. First, 
continuation was based on a program’s operation for one additional year independent of federal 
funds. The continuation of these workplace literacy programs beyond this point cannot be 



8 Site 1 confronted the additional difficulty of limited employer involvement during the course 
of the NWLP grant. Since it emphasized employees’ finding better jobs, current employers were 
not willing to finance continued services and new employers were not willing to support a pool of 
potential hires. 
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determined from this study. Second, many programs that continued were changing in content and 
form. Some, like Site 2, planned to offer shorter duration courses in the future. A Site 4 employer 
planned, in collaboration with the community college, to institute an on-the-job college in place of 
the workplace literacy program. As programs became free of NWLP requirements, we noted three 
common types of changes: 

• an emphasis on content tied to job skills and job training 

• courses that ran for fewer hours and involved more workshops and individualized 
learning that was off-the-clock 

• inclusion of personal interest courses to help workers manage areas such as stress, 
retirement planning, and preparation of taxes. 

It is difficult to interpret and predict the significance of these program alterations for the NWLP 
model of workplace literacy. We broadly interpreted the NWLP goal of “integration of workplace 
literacy services into the long-term planning of partner organizations” (CFR 472.22. h) to include 
any local plans that called for continuing with some form of job-oriented basic skills education, 
whether of short duration or inclusive of more job training. Initial indications suggest that over time 
we may see the basic skills aspects of the NWLP model of work-based education less pronounced 
in the programs that continue solely with company financing. 

D. LESSONS ABOUT STATE WORKPLACE LITERACY INFRASTRUCTURE 

We conducted case studies of state-level infrastructure in two sites that were developing 
workplace literacy capacity through their system of two-year community/technical colleges (Site 4 
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and Site 5). 9 Figure II. 2 shows the state and local organizational structure of these two sites. We 
chose to focus on the experience of these sites because community or technical colleges were the 
most common education providers among all NWLP grants, serving as education partners in 49 
percent of the NWLP partnerships funded in 1994. 10 

Many community/technical colleges pursue workplace literacy initiatives independent of an 
organizational superstructure. Indeed, many cherish autonomy from such higher levels of authority. 
Nonetheless, linking these stand-alone efforts in an organized system offers potential advantages for 
improved efficiency and expertise in meeting employers’ new or changing literacy requirements. 
Researchers recently have underscored the pivotal role of regional community/technical colleges in 
providing occupationally relevant education through the formation of “adult learning communities” 
to replace the currently fragmented patchwork of post-high school educational options facing adults 
(Grubb 1996). Clearly, as established institutions with a mission increasingly oriented toward 



’Several states have contributed funds to support workplace literacy services. The sources they 
have used include tax credits, federal funds distributed to states under the Adult Education Act, state 
funds for adult basic education, and community/technical college funds. Anecdotal information 
indicates that the amounts directed by state policy makers to workplace literacy vary considerably. 
The wisdom of such public subsidies has been a major subject of debate. Proponents argue that 
without subsidies few employers will invest in workplace literacy programs because of high start-up 
costs and uncertainty of a payoff. Opponents cite the problem of subsidies serving as a windfall for 
some employers who independently would have invested in these programs (Osterman and Batt 
1993). 

,0 Other education partners included four-year postsecondary institutions, state education 
agencies, local education agencies, and community-based or manpower development organizations. 
None of these partner categories was as numerous as community colleges. 
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occupational education, the system of community/technical colleges in many states offers an 
attractive platform for infrastructure to support workplace literacy." 

1. Differences in Sites’ Scale and Approach to Infrastructure Development 

Although the state-level sites shared an emphasis on community or technical colleges, they 
differed in two important ways: (1) the scale of operations and (2) the approach to building an 
infrastructure. 

a. Scale of Operations 

As Figure II. 2 and the site profiles at the beginning of this chapter demonstrate, the workplace 
literacy program in Site 5 operated on a much larger scale than that in Site 4. Organizationally, the 
Site 5 partnership included four state-level partners, 1 1 colleges, and 20 employers. In contrast, Site 
4 had one state agency partner and a network of five education providers (four colleges and one 
community-based organization) collaborating with ten employers. Of course, the presence of more 
employers in Site 5 meant more local staff were involved than in Site 4; an even more significant 
difference was the configuration of staff. The organizational structure in Site 5 called for workplace 
literacy coordinators at each college whose role was to develop employer programs and to oversee 
instructors assigned to specific employers. Site 4’s smaller scale was associated with a simpler role 
configuration: instructors from the college level served as lead teachers whose role was to develop 
and deliver services at employer sites, sometimes in conjunction with another instructor. 




1 'Three of the seven NWLP state-level partnerships worked primarily with community colleges. 
Other strategies placed the state education agency in collaboration with union and employer partners. 
The education agency brokered arrangements, including finding appropriately trained instructors to 
hire. One state oversaw a network of diverse providers, which included some independent 
community colleges and four-year colleges. Future studies could profitably explore these alternative 
approaches. n r 
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Three features account for this difference in the scale of workplace literacy operations: (1) Site 
5 has received NWLP grants for almost twice as many years as Site 4; (2) throughout these years, 
Site 5 also has operated a state-funded competitive workplace literacy grants program similar in 
approach to its current federal NWLP grant; 12 and (3) Site 5 has focused on refining a process for 
local staff to use to expand literacy programs to new employer sites, whereas Site 4 concentrated on 
assembling a coalition of experienced workplace instructors from a few colleges to develop a 
curricular model addressing the literacy skills associated with TQM reforms. In summary, Site 4 
is best seen as a fledgling system of infrastructure. In contrast, operations in Site 5 have matured 
into a broad statewide system of infrastructure oriented to workplace literacy. 

b. Curriculum Versus Process Approaches to Infrastructure Development 

While differences in the scale and maturity of the two sites are important, equally important are 
differences in their approaches to developing infrastructure for workplace literacy programs. Site 
4 has pursued a curricular approach to infrastructure. After using an initial NWLP grant to build the 
expertise of instructors and to connect these instructors and their colleges with businesses in their 
areas, Site 4 has focused on constructing a curricular template of core literacy skills and lessons tied 
to TQM. The goal has been to develop a tool to help instructors efficiently identify and formulate 
lessons addressing the core literacy skills necessary to applying TQM throughout the workplace. 
The core literacy skills were to support requirements for employees to be proficient at team building, 
problem-solving, decision-making, and communications. 

12 These funds from the state technical college board are allocated at the board’s discretion to 
workplace literacy grants and have averaged around $500,000 per year. Employers in the state who 
participate in the state grant program receive funds for up to three years from one or the other source. 
Currently there are few differences between the two programs, with the exceptions that most of the 
state-level activities are federally supported and state funds permit the purchase of equipment related 
to workplace literacy services at local sites. 
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Developing a common process for instituting successful workplace literacy programs has guided 
Site 5’s approach to infrastructure. Although Site 5 has shared information about promising 
curricula used by its workplace literacy instructors, it also has invested heavily in developing 
procedural manuals, training, and providing technical support on a broad range of processes. These 
processes include conducting analyses of the literacy skills of jobs, instituting steering committees 
and teams of workers to serve as peer advisors at each work site, establishing effective methods for 
recruitment of employees, and generally introducing and promoting workplace literacy instruction 
to employers and employees. Furthermore, Site 5 obligates employers to contribute an increased 
fraction of the cost for each of the three years they participate in either the federal or state-funded 
grants. 

The contrast between an emphasis on process and on curriculum offers insights into workplace 
literacy infrastructure. The process approach appeared to generate fewer implementation difficulties 
for instructors than did the curricular approach. The instructors in Site 4 believed the curricular 
template, if developed as planned, would interfere with constructing a curriculum responsive to 
employers’ unique requirements and contexts. The instructors’ concerns led to a revised plan to 
assemble such a template only after the work-site customized curricula were in place. Since the state 
team has yet to complete the template, however, its application to new settings remains to be tested. 
Overall, the experience of the two sites suggests that core curricula may be a more difficult part of 
infrastructure to put in place than processes that surround local development of customized curricula. 




52 



2. Common Infrastructure Functions Performed by State-Level Partners 

Despite the differences in approach, the two state-level partnerships performed similar functions 
to foster infrastructure. We suspect that these functions may be key to building statewide expertise 
and capacity to broaden the reach of workplace literacy. 



• Definition and Dissemination of a Common Model of Workplace Literacy. Each site 
articulated a model to guide local partners. Site 4 used a two-part formula: instruction 
(including ESL instruction) to address the basic literacy skills needed in workplaces 
implementing TQM reforms and governing task forces composed of key stakeholders 
(for example, managers, supervisors, employees, and the instructor). The model in Site 
5 contained several elements: governance by a workplace steering committee, voluntary 
participation for employees, instruction that was self-paced and individualized, a team 
of peer advisors, employers’ sharing in cost based on cash, and confidentiality of 
employee data. The model served as a guide to what the state partners considered best 
practice. 

• Professional Development of Instructors. Throughout the year both sites regularly 
convened instructors to share effective approaches and obtain suggestions for remedying 
problems. Additionally, they used mentor relationships to develop new instructors. Site 

4 relied on a lead teacher to perform this function; Site 5 assigned this role to the 
workplace literacy coordinators at each college. A variety of manuals listing effective 
practices and guidebooks also were given to all instructors in Site 5. 

• Network Development and Information Exchange. Both sites cultivated networks of 
professional staff. Site 5 often expanded the workplace literacy networks to include 
peer advisors from various local plants and employer representatives on steering 
committees. The site provided funds for travel to state meetings and employer sites 
where local staff could observe instructional techniques applied in real settings. The 
state-level partners in both sites were key contacts for information about workplace 
literacy programming, relevant materials, and resources in and outside of the state. 

• Generation of Participation and Assessment Data. Data were gathered by the state 
partnership in both sites, but Site 5 was more systematic in its collection and 
dissemination of data from the local sites. The federal NWLP requirements for data 
were a major catalyst behind the partnerships’ generation of data from local work sites, 
although the state workplace literacy grants program in Site 5 reinforced that 
partnership’s insistence on data about numbers of participants in various types of 
instruction, their demographic characteristics, instructional hours, expenditures, 
numbers of job skills analyses completed, and reports of outcomes. Both Site 4 and Site 

5 encouraged all sites to gather assessment data on individual participants but did not 
require specific assessments. Both state partnerships indicate the collection of data has 
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been important for informing employers, colleges, and state officials about workplace 
literacy. 

• Assistance in Problem-Solving and Conflict Resolution. Access to an individual with 
expertise to solve problems and defuse conflicts was available through the state partners 
in both partnerships. The regular meetings of instructors and, at Site 5, coordinators 
supplied a forum for surfacing many such issues. State-level staff were also accessible 
through phone contacts and frequent visits to the sites. Site 5 also provided technical 
support through the college coordinators, who typically maintained weekly contact with 
the instructors at the work sites. Problems such as handling labor/management 
disagreements, gaining front-line supervisors’ support, helping employers articulate a 
realistic set of program goals, and recruiting employees typically prompted 
communication and collaboration. 



3. Importance of Decentralization, Flexibility, and Support from Local Colleges 

Both state-level partnerships in this study stressed a decentralized form of organization. It is 
not surprising that these state initiatives to develop workplace literacy capacity operated in a 
decentralized mode, given the significant level of autonomy of community /technical colleges. The 
job-specific, workplace-specific principles that are central to workplace literacy as a field also 
greatly reinforced decentralized arrangements. The sites’ decentralization has meant that colleges 
were free to decide whether to participate in workplace literacy programs. Such decisions as 
instructor hiring, employer recruitment, and organizational location of the workplace literacy staff 
in the colleges also were left to local determination. 

The flexibility of a decentralized organizational structure clearly was evident in the adaptations 
that Site 4 and Site 5 have allowed local staff in their application of the state-defined model of 
workplace literacy. As noted previously, Site 4’s plan for a curricular template was revised to allow 
instructors to first develop a literacy curriculum focused on individual employers. Site 5 also has 
given wide berth to local colleges to alter various features of the state’s defined model for a 
workplace literacy program. Flexibility for local sites, however, did not undermine either sites’ 
promotion of a model of workplace literacy. Instead, flexibility appeared to be a practical necessity 
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as partnerships recognized the limits of a uniform model in diverse work environments. As one state 
staff member observed, “If we learned anything, it’s that in order to be effective you have to find 
ways to work with businesses that they are comfortable with and that fit their environment.” 

For colleges to play a key role in providing the capacity to deliver quality workplace literacy 
programs, they must invest as institutions in building and maintaining such a capability. Simply 
serving as the organizational affiliation for entrepreneurial instructors with workplace literacy 
expertise may be a step toward developing a college’s ability to work with nearby employers, but 
it is unlikely to foster the long-term capacity to deliver quality instructional programs in the 
workplace. One reason is that colleges that simply provide an affiliation for entrepreneurial staff risk 
losing those staff. In many local sites, successful instructors often were recruited to work directly 
for employers, resulting in the colleges’ loss of capacity to serve other employers. To build long- 
term capacity, the colleges themselves have to offer incentives for instructors to remain. They can 
do so by contributing resources and leadership that elevate and integrate workplace literacy services 
within the overall mission of the school. 

The organizational placement of workplace literacy staff within the college may be a sign of 
a college’s commitment to building capacity for workplace literacy services. In both state-level 
partnerships, some colleges had placed workplace literacy within divisions that focused on business 
outreach or workforce development instead of basic skills or continuing education for adult learners. 
This placement overcame some reported reluctance within the colleges’ traditional divisions to give 
instructors and coordinators the support and flexibility necessary for establishing responsive working 
relationships with employers. Yet, while this type of placement may have proven beneficial to the 
development of workplace literacy expertise at some colleges, it was by no means clear that the 
organizational location itself is critical. Support within the college from division heads and the 
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president seems to be the crucial concern— and that support may or may not be reflected in a specific 
organizational placement. 13 

4. Institutionalization of Statewide Infrastructure 

The continuation of statewide infrastructure— as distinct from local workplace literacy services- 
is largely an issue for the educational institutions and state organizations in which the infrastructure 
has been developed. Earlier in this chapter we discussed the continuity of workplace literacy 
beyond the terms of the federal NWLP grants. In this section, we focus on the prospects for 
continuation of workplace literacy infrastructure at the community/technical colleges and state 
organizations in the two state-level partnerships we studied. 

Based on these two sites, we surmise that institutionalization and growth of infrastructure at the 
colleges will depend on a consistent level of employer demand for workplace literacy program 
support, either on a fee-for-service basis or as subsidized by state funds or tax incentives in 
conjunction with the presence of a faculty whose primary identification, despite their part-time or 
adjunct status, is with the college. The continuation of infrastructure at the state level will depend 
on the availability of funds from state or possibly other sources to support the technical assistance 
roles and activities that state coordinators have been performing. The components of infrastructure 
at the state level put most at risk by the termination of federal support are areas such as travel, in- 
person training, and trouble-shooting that state policy makers traditionally have been reluctant to 
fund. 
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13 Grubb (1996) also has observed that while sometimes accommodating to employers, 
organizationally placing contract education and employer outreach activities— and presumably 
workplace literacy— at a distance from the core divisions of the college can limit the positive 
influences within the college from closer working relationships with employers. 
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a. Infrastructure in the Colleges 

Within Site 5, the availability of expert staff who promote and work with employers to deliver 
customized workplace literacy services will probably continue. The presence of workplace literacy 
coordinators at most colleges in the state is a major factor in the expected continuity of infrastructure 
at the college level. These individuals’ roles are secured for at least the next few years as a result 
of their responsibilities with the state grant funds for developing new employer-sponsored programs. 
The state plans to shift its cost-sharing grants from three to two years to spread state funds and thus 
open more learning centers than in the past. 

The college coordinators’ and instructors’ roles also are secured by their continuing role with 
previously funded employers whose workplace literacy activities will carry forward through 
contracts with the colleges for specific educational or technical workshops. Over half of the 
employers that participated in Site 5’s NWLP grant, for example, plan to extend their workplace 
literacy efforts through financial arrangements with the colleges. This is noteworthy given the 
higher cost that employers typically encounter when using the college-hired instead of in-house 
instructors. Several previous employers also plan to contract with local colleges for specific 
educational programs or workshops; in fact, colleges in Site 5 experienced considerable growth in 
contracting with employers in the past several years. 

The prospects for continuity of infrastructure at the community/technical colleges in Site 4 are 
less clear. Several instructors in Site 4 are now employed by the companies they served. Moreover, 
the lack of formally designated coordinator roles in the affiliated colleges and the absence of any on- 
going funding to stimulate employers’ investment leaves the future of infrastructure within Site 4 
colleges quite uncertain and dependent on the colleges’ acquisition of financial support for 
workplace literacy services from new and former employers. 



b. Infrastructure at the State Level 



The state-level team in Site 5 is likely to remain in place and active in guiding the development 
of workplace literacy programs in partnership with the local colleges, but the extent and nature of 
their activities is likely to change. The state funds for starting new workplace literacy programs will 
provide a focus for maintaining connections with the college coordinators. A variety of hard-copy 
manuals and guidebooks produced under previous NWLP grants will remain available for local use. 
The development of future guides and consultative assistance for local colleges will probably 
continue but to a more limited extent since federal NWLP funds have been the mainstay of support 
for these efforts. Similarly, travel for meetings, site visits, and training sessions will probably be 
more restricted in the future. Traditionally, state legislatures generally have little appetite for 
spending in these areas and have limited their state agencies’ authority to spend accordingly. 

Site 4’s prospects for maintaining a state-level infrastructure in the form of a dedicated staff 
coordinator with networking capability are tied to the state board’s ability to find alternative sources 
of project support. Although the state legislature has considered three workplace literacy bills in past 
years, this legislation has failed to move ahead. Nevertheless, a state workforce board recently 
endorsed efforts to find alternative support for the workplace literacy project’s continuation. The 
outcome of discussions at the state level is far from certain, but at this point there are no sustaining 
funds in clear sight. 
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III. THE IMPACT OF WORKPLACE LITERACY 



The national evaluation’s impact study had two goals: (1) to assess the effectiveness of 
workplace literacy on workers’ literacy and employment-related outcomes and (2) to demonstrate 
how rigorous experimental designs, such as random assignment, could be implemented to assess 
program impacts. While many of the impacts of workplace literacy occur in one site, the results of 
our evaluation show that this approach has some modest short-term impacts in a variety of areas 
including workers’ career and educational plans, literacy skills, literacy activities outside of work, 
job performance, and other employment-related outcomes. Furthermore, this evaluation 
demonstrates that it is possible to evaluate the effectiveness of workplace literacy using random- 
assignment designs, although given the context of program operations this can be very challenging. 

In Section A of this chapter we explain the procedures used to set up the random assignment 
design. Section B describes the kinds of data used in the impact analyses and variations in data 
collection that occurred across sites. In Section C we present our findings on the impacts of 
workplace literacy. Finally, in Section D we discuss differences in impacts observed between the 
three impact-study sites and the potential importance of instructional time. Lessons learned from 
our implementation of a random-assignment design to evaluate program impacts are discussed in 
Chapter IV. 



A. RANDOM SELECTION OF WORKERS FOR WORKPLACE LITERACY COURSES 



Random selection of workers for the treatment and control groups operated similarly in each 
site. To accommodate the experimental design, staff in each site were encouraged to recruit twice 
as many eligible workers (applicants) as there were course openings. Only workers who had not yet 
participated in an NWLP course were eligible for random assignment. Next, program staff typically 
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assessed their applicants to determine what course was best suited to meet their needs, considering 
factors such as standardized literacy test scores, motivation, and work schedule. The staff then 
provided lists of course applicants to MPR. 

Next, for each course MPR randomly assigned eligible applicants to either a treatment group, 
which was allowed to enroll in a workplace literacy course right away, or a control group, which 
would not be eligible to participate at least until the current course cycle ended. 1 MPR then notified 
the impact sites about the selection status of each eligible worker, and project staff accordingly 
notified the applicants whether or not they had been admitted to a course. 

This process was repeated for each course cycle at each site, from September 1995 through 
February 1997. The sites generally offered courses beginning at three or four times during this 18- 
month period. MPR’s sample incorporated workers randomly assigned throughout this rolling 
enrollment period. Table III. 1 shows the sizes of the samples that we constructed using this 
approach. 2 

Each site experienced some difficulty carrying out the actual implementation of the proposed 
research design. 3 One significant issue for the sites concerned the number of course cycles that 



’The actual waiting period varied between sites, as is described later in this chapter. To avoid 
the possibility that workers would be randomly prohibited from enrolling in workplace literacy more 
than once, those assigned to the control group in one course cycle were excluded from the random 
assignment pool in all subsequent course cycles; that is, they were allowed to take courses later 
(subject to staff approval), without going through the random-assignment process. 

2 Because some workers assigned to the control groups were not served at all during the time 
period under study, and because sites served some workers who were not subject to the random- 
assignment experiment, the sum of the number of workers in the two columns in Table III. 1 does 
not match the total number of learners served as reported in Chapter II. In addition, this table does 
not indicate the number of study participants included in the impact analyses; missing data and other 
issues reduced these numbers for various analyses. 

3 Appendix B describes in more detail the implementation issues associated with each of the 
sites. 



TABLE III.l 



NUMBER OF WORKERS RANDOMLY ASSIGNED TO 
TREATMENT AND CONTROL GROUPS, BY SITE 



Impact Site 


• Treatment Group 


; r ; : Control Group 


Site 1 


260 


149 


Site 2 


93 


93 


Site 3 


195 


153 



workers would remain in the control group. The design initially proposed by MPR and ED required 
that workers randomly selected for the control group be held back from taking courses for two course 
cycles. For Site 1 this would have meant that about half of the interested workers could not attend 
NWLP courses for 34-36 weeks; in Site 2 the delay would have been 24-28 weeks; and in Site 3, 16- 
18 weeks. Before implementing the design, Site 1 ’s project director requested that we shorten their 
delay period because many workers would not begin participating in NWLP for close to a year, and 
enrolling enough learners to meet the goals outlined in the site’s grant proposal would be difficult. 
In response to these concerns, we adjusted the design and required a delay of only one course cycle 
for Site 1 . Also, after two rounds of selecting workers in Site 3, the project director asked if they 
too could reduce the delay for control-group workers to one course cycle instead of two, because the 
site’s employers wanted their employees to obtain NWLP services more quickly. We agreed to the 
change after consulting with ED staff. 

A second design component that proved problematic was the use of standardized literacy 
assessments. Site 3 took until several months after random assignment began to select a test, 
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meaning that not all their study participants received a baseline assessment. 4 After a literacy test had 
been selected, many workers in Site 3 may have been given the wrong level of the pretest because 
the site staff did not use a short locator test supplied by the publisher. We did not identify this 
discrepancy until after the second round of random selection. Because of these problems, we 
excluded this pretest information from our analyses. 

The third design component that initially proved difficult, again for Site 3, concerned 
maintaining the integrity of the treatment/control group design. A few months into the study we 
learned that in Site 3 many workers in the control group had taken NWLP courses after one course 
cycle instead of two course cycles, as originally planned. As many as 73 control group members 
from the first two course cycles may have taken courses earlier than they should have. To ensure 
that we obtained unbiased estimates of program impacts, we excluded all Site 3 study participants 
(workers in both the treatment and control group were excluded) who were selected during the first 
and second rounds of random assignment. 

By removing from the analysis sample all workers in Site 3 who were in the first and second 
rounds of randomization, we reduced the size of the treatment group from 195 to 1 18 workers and 
the control group from 153 to 80 workers. Although we maintained the integrity of the random 
assignment design by excluding all members of the treatment and control group who were selected 
in the first two rounds of random selection, we also reduced our ability to detect program impacts 
for this site. With the full sample of treatments and controls we would expect, for example, that we 
could detect an impact of 10 percentage points or larger; with the smaller sample size we could 



4 Site 3 initially used the Adult Language Assessment Scale (LAS), an instrument that measures 
oral English skills, but later added the Comprehensive Adult Student Assessment System (CASAS) 
listening and reading test to better assess learners’ acquisition of a broader range of English language 
skills. 
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detect an impact of 14 percentage points or larger— about the same size of impact that we can detect 
with the sample of workers in Site 2. This means that if we computed an impact of 13 percentage 
points, we would conclude that the impact was not significant using standard statistical criteria. 5 
Even though we could not measure impacts as precisely with the smaller sample sizes, meaningful 
impacts could still be detected in Site 3. 

While each site experienced some difficulty in setting up random assignment, we note that none 
of the issues described above jeopardized our ability to make causal statements concerning the effect 
of workplace literacy on worker outcomes. Each of our adjustments to the random assignment 
design preserved the character of the treatment group and control group so that workers in the two 
groups were statistically equivalent before the partnerships offered workplace literacy services. 
Because the only difference between the groups should be the offer of services, we are confident that 
the observed differences in outcomes between the treatment group and the control group arose 
because of workplace literacy and not other factors such as personal characteristics or changes in the 
workplace. 

B. OUTCOME MEASURES AND DATA COLLECTION 

Our evaluation utilized information on a broad range of literacy-related outcomes, obtained 
through both standardized tests and a variety of other assessment instruments. As Mikulecky and 
Lloyd note, "Workplace literacy program impact is best measured using a mixture of standardized 
tests and custom-designed instruments. Standardized tests provide useful information about general 
reading ability, but may be misleading with regard to workplace literacy skills” (1993:33). Using 



5 We assumed when making this calculation that a one-tailed test of statistical significance is 
used, the probability of rejecting the hypothesis of no impact by chance along is .10, and the 
probability of rejecting the hypothesis of no impact when there is an impact in the population is .80. 



a broad range of outcomes was particularly important for this evaluation because the three impact 
sites differed in their curricular emphasis and the kinds of workers they serve (as described in 
Chapter II). 

1. Collection of Baseline Data 

When workers at the three sites applied to participate in workplace literacy, NWLP staff gave 
them a baseline questionnaire and a standardized literacy assessment. The questionnaire asked 
workers about their background characteristics, literacy behaviors at home and work, and abilities 
to perform various literacy and work-related skills. The standardized tests were used to assess 
workers’ abilities in basic skills such as reading and writing English, and math. We allowed each 
site to choose a test that best fit its curriculum. 6 For Sites 2 and 3, we also sent questionnaires to 
workers’ supervisors, asking them to rate how well study participants performed various job-related 
tasks. 7 

2. Collection of Follow-Up Data 

After each course ended, workers in the treatment group and the control group each completed 
follow-up questionnaires generally similar to the baseline instruments, but with additional questions 
about recent job experiences and future educational plans. Both groups were again given a 
standardized literacy assessment. At Sites 2 and 3, supervisors filled out a follow-up questionnaire 
identical to the one from baseline. We also sent questionnaires to employers at Sites 2 and 3, asking 



6 Site 1 chose to continue using a standardized literacy assessment that they were already using, 
the John Test; Sites 2 and 3 both selected new tests, in each case they used versions of CASAS. Site 
2 used reading and math tests and Site 3 used listening and reading tests. 

’Because Site 1 did not maintain close working relationships with employers, we did not send 
questionnaires to employees’ supervisors. 
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about workers’ current and past wages, performance bonuses, attendance, and job-related accidents. 8 
Table III.2 indicates the types of data collection instruments used in each site. (Copies of the in- 
depth study questionnaires are provided in Appendix G and response rates for all the data collection 
instruments are shown in Appendix E.) 

After each course ended, NWLP staff also recorded the kinds of courses that learners took and 
the number of hours of instruction they received in these courses. This information was entered into 
the National Workplace Literacy Information System (NWLIS) and later merged with the other data 
collected as part of the impact study. Final data collection took place in July 1997. 

C. PROGRAM IMPACTS 

Literacy instruction linked closely to the workplace can lead to important changes in workers. 
Our analyses suggest that workplace literacy can affect workers’ career and educational plans, 
improve their literacy skills and job performance, and change their literacy habits at home. 
However, in most cases the impacts we found are small and seldom occurred in more than one site. 

This section focuses on impacts derived from the random-assignment experiment. These 
estimates of program impact are based on a comparison of outcomes for the treatment group and the 
control group. The difference between the two groups on a given outcome (for example, the 
percentage who change their educational plans) provides an estimate of the program’s impact; that 
is, an estimate of how much, on average, workplace literacy affected worker outcomes. This 
approach for computing program impacts produces the most credible evidence on how linking 
literacy instruction to the workplace affects workers’ literacy skills and job performance. The only 
difference between the groups after random assignment should be the receipt of NWLP services 



8 Because Site 1 workers had numerous current employers whose names and addresses were not 
maintained by the partnership, we did not send questionnaires to the workers’ employers. 
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TABLE III. 2 



OVERVIEW OF DATA COLLECTION FOR IDS IMPACT STUDY 



Instrument 


12'" ./j Site 


• / Site 


III ||it|:|l:J. : ;-'i 


Worker baseline questionnaire (“Baseline Data Form”) 


Yes 


Yes 


Yes 


Worker follow-up questionnaire (“First Follow-Up Form: 
Participants” or “First Follow-Up Form: Waiting List”) 


Yes 


Yes 


Yes 


"Supervisor Form" 3 


No 


Yes 


Yes 


“Employer Form” 


No 


Yes 


Yes 


Standardized Literacy Assessment* 3 

ar* i i _ . i- j #• « 


John Test (a test 
of oral English 
proficiency) 


CASAS 
reading and 
math test 


CASAS 
listening and 
reading test 



a Form used at both baseline and follow up. 

^Not all study participants at a given site were tested at both baseline and follow up. 



because the two groups are, in theory, statistically equivalent at the time of assignment, thus 
reducing the possibility that selection effects may bias our estimates of program impact. 9 

The estimates of program impact presented in this report show the effect of being offered the 
opportunity to attend NWLP courses; that is, the impact of workplace literacy on all workers 
randomly selected to participate in courses. These impact estimates combine the effects of two 
processes: (1) showing up for at least one NWLP course and (2) workers continuing to participate 
in the course. Some workers selected for participation chose not to attend any courses. This was 
particularly noteworthy in Site 3 where data reporting problems combined with workers’ decisions 



Selection effects, however, may still occur because of (1) differences between the two groups 
in their response patterns during follow-up data collection or (2) chance differences between the two 
groups at the time of random selection. Appendix E shows the characteristics of respondents and 
nonrespondents. Characteristics of the workplace literacy and control groups at baseline are 
presented in Appendix F. To deal with chance differences between the workplace literacy and 
control groups, we used an analytic model that statistically adjusts for differences in the observed 
background characteristics that may exist between the two groups before the intervention. A 
complete description of this procedure is also provided in Appendix F. 
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not to participate created a large fraction of what may be no-shows. According to the attendance 
data submitted by the sites, in Site 1 about 95 percent of the workers randomly selected for NWLP 
courses attended at least one class session; in Site 2 about 7 5 percent attended at least one class, in 
Site 3 about 50 percent of the workers selected for the treatment group attended at least one class 
session. Because of possible discrepancies in sites’ recording of attendance data, we decided to 
focus the impact analyses on all workers rather than for those who attended courses. 10 By focusing 
the analyses on the impact of being given the opportunity to take NWLP courses, the treatment 
in this study is defined more broadly than simply what instructors accomplish once workers come 
to a course; it also includes instructors’ efforts in getting workers to show-up for courses. Both 
aspects of workplace literacy programs are critical to include, since a program that fails to attract a 
sizable fraction of targeted workers into courses could not be termed fully effective. 

1. A Context for Interpreting Findings 

The three sites included in the impact study differ along several dimensions that may affect how 
we interpret the findings. Keeping these distinctions in mind when reviewing the results is useful. 11 
For example, differences in course emphasis may suggest to some readers that certain outcomes are 



10 To assess how sensitive our conclusions were to this decision, we computed impact estimates 
for both groups; in so doing, we found only a few differences (the approach we used to adjust for 
some workers in the treatment group not attending classes is described in Angrist, Imbens, and 
Rubin 1996). Besides the impacts reported in this chapter for all workers, we observed for the 
course participants in Site 1, an impact of workplace literacy on workers’ reports about receiving 
instructions in English while on the job; for Site 2 we found an impact on supervisors ratings about 
workers following verbal instructions; for Site 3 we observed an impact on supervisors’ ratings about 
workers asking coworkers/customers if they needed help and workers improvement in completing 
tasks on time. Also, after we adjusted for workers never showing up for classes, the impact of 
workplace literacy on changes in educational and career goals was no longer significant for Site 3. 




11 Additional description of site and worker characteristics is presented in Chapter II and 



Appendices A, B, and F. 
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more relevant than others for concluding a project achieved success. Some of the differences 
between sites include: 



• Experience Operating a Workplace Literacy Program. Site 1 had more experience 
offering workplace literacy instruction than either Site 2 or Site 3; Site 1 had operated 
under three NWLP grants while Sites 2 and 3 had only received a grant in 1994. 

• Course Emphasis. Courses offered in Sites 1 and 3 had an ESL emphasis, and Site 2 
emphasized improving basic skills. 

• A verage Amount of Instruction . The typical amount of instruction received by workers 
for the first course they attended in the three sites differs. For example, for the first 
course taken by employees, the typical hours of instruction received in Sites 1, 2, and 
3 were 43 hours, 31 hours, and 19 hours, respectively. 

• Population Served. Most of the workers in Sites 1 and 3 were foreign bom; only a few 
of the employees in Site 2 were foreign bom. In Site 1 about one-quarter of the learners 
had less than 9 years of schooling and in Site 3 almost half had attended school for less 
than 9 years; in Site 2 almost 90 percent of the employees had attended school for 12 or 
more years. Nevertheless, it was clear that Site 2 workers had considerable skill deficits 
despite their longer exposure to formal schooling. 

• Literacy Skills Needed on the Job. Workers in Site 2 were more likely to report that 
they needed to use English or math skills on the job than workers in either Site 1 or Site 
3. 



Another relevant distinction when examining the findings concerns short-term and long-term 
outcomes. In each site, we contacted workers immediately after a course was completed. This 
means we usually examined the impact of the workplace program over a period of about three 
months and assessed the contribution of taking on NWLP course. 12 The occurrence of long-term 
outcomes such as changes in employment status, job-related responsibilities, wages, and fringe 
benefits were improbable for most respondents within this short time frame. 




l2 Nationally, only about 25 percent of learners took two or more courses. 
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2. Results From Analyses Based on Random-Assignment Design 

a. Workplace Literacy Improved Workers’ Self-Reported Literacy Abilities and 

Standardized Literacy Test Scores 

Partnerships selected for the NWLP program were supposed to be committed to delivering job- 
linked basic skills instruction. The NWLP defined basic skills broadly to include reading, math, 
English proficiency, communication, reasoning, problem-solving and the completing of a high 
school degree or its equivalent. Our data collection forms focused directly on these issues and asked 
workers to rate their own ability to read, speak, write, and understand English; to use math; and to 
solve problems or use reasoning. 13 We also assessed workers’ literacy abilities through standardized 
literacy tests, comparing how workers in the treatment group scored at follow up with how they 
would have been expected to score in the absence of the workplace literacy course. The results show 
that workplace literacy had small or modest impacts on workers’ self-assessments in two sites and 
a very large impact on standardized literacy test scores in one site. Very few workers completed a 
GED between the time they started a course and the end of the course and there were too few 
instances to judge whether workplace literacy affected this outcome. 14 

In Site 1 workers’ self-assessments revealed program impacts on all six of the literacy-related 
skills we asked about; in Site 2 there were impacts in two areas— workers’ abilities to read and 



13 The four-point response scale ranged from “poor” to “excellent.” 



14 For all statistical tests of no impact we used a one-tailed test and set the probability of rejecting 
the hypotheses of no impact by chance at .10. By using a one-tailed test we treat both negative 
impacts and zero impacts the same; that is, we only distinguish between positive and nonpositive 
impacts. 
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understand English (see Table III. 3). 15 No impacts were observed in Site 3. For both sites with 
impacts, the size of the impacts ranged from about .2 to .3 standard deviations. An impact of .2 to 
.3 standard deviations shows that about 60 percent of the workers who were selected to attend an 
NWLP course had self-reported literacy skills that were above the average worker in the control 
group. 16 These findings suggest, for example, that after taking a course with an ESL emphasis for 
about 4-5 months in Site 1, 10 percent of the course takers reported they had better ability to 
understand, speak, read, and write English. Furthermore, about 15 percent of the workers had better 
math skills and about 10 percent were better able to solve problems. 

We also found an impact on standardized literacy test scores in Site 1 among workers in ESL 
classes, but no impacts at the other sites. 17 The impact in Site 1 is very large-almost 1.2 standard 
deviations, which is about four to five times larger than the impacts based on self-reports of literacy 
skills. An impact this large suggests that more than 85 percent of the workers who attended a 
workplace literacy course scored higher than the typical worker in the control group. 



l5 Table III.3 and all subsequent tables in this chapter present impact results in terms of standard 
deviation units (that is, impact estimates divided by the pooled standard deviation for a variable), 
also referred to as effect estimates (Cohen 1988; Hedges and Olkin 1985). We use this approach for 
two reasons: first to help us interpret the size of the impact when variables are measured with 
ordinal scales (it can be difficult, for example, to interpret small differences on a four-point scale 
where 1 = “poor,” 2 = “fair,” etc.); and second, so that we can compare items measured with different 
scales and metrics. Impacts in the range of .2 to .3 standard deviations are usually classified as 
small; impacts of roughly .5 standard deviations are generally considered medium-sized; and those 
around .8 or higher are considered large (Cohen 1988). 

16 If there were no impact, we would expect that 50 percent of the workers in the control group 
would have skills below the average worker in the treatment group. 

17 In Site 1, students who were assigned to the ESL courses were given the John Test and those 
assigned to the Basic Skills courses were given a standardized basic skills test; all workers in the 
control group were given the John Test. To compute the impact of workplace literacy on workers’ 
standardized test scores, we limited the sample to only those who were eligible to take an ESL 
course and their corresponding control group formed at the time of random assignment. 



TABLE III. 3 



PROGRAM IMPACTS ON LITERACY SKILLS, 
IN STANDARD DEVIATION UNITS 



Outcome Jf ||§ 




7’;. Site 2 .nJm 


Site 3 


Workers’ self-rated ability to: 








read English 


.34* 


.23* 


.22 


understand English 


.28* 


.25* 


.03 


speak English 


.23* 


.09 


.06 


write English 


.18* 


-.06 


.13 


use math 


.38* 


.12 


.08 


solve problems/use reasoning 


.17* 


.10 


-.15 


Standardized Literacy Assessment 


1.15* a 


-,00 b 


reading: -.24 c 






listening: -.15 c 


Received a GED 


-.29 (-9) d 


-.41 (-9) 


NA e 



*Significant at the .10 level using a one-tailed test. 

“Only treatment-group members in ESL courses and their control group were tested at follow up. 
b Only treatment-group members in math courses and their control group were tested at follow up. 
C A11 study participants tested at follow up. 

d Impact expressed in standard deviation units; impact as a percentage point difference is in the 

parentheses. 

e Could not estimate. 



b. Workplace Literacy Had Small Impacts on Workers’ Career and Education Plans 

To determine if workplace literacy might affect workers’ career and educational plans, at follow 
up we asked all study participants whether they had recently changed their educational or career 
goals, whether or not they planned to take certain courses in the future, and whether they had already 
signed up for another course of some type. Our analysis shows that the program did have some 
impacts on workers’ plans, but these effects were generally small and not widespread across the 
sites. 
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At Site 1 we found an impact of 1 1 percentage points on workers changing their career or 
educational goals (see Table III.4). 18 This indicates that the percentage of workers in the treatment 
group who changed their goals was 1 1 points greater than the percentage of workers in the control 
group who changed their plans (in this case, 52 percent versus 41 percent, respectively). Site 3 also 
had an impact on this outcome, but it was only 6 percentage points. 

In terms of future course-taking plans, at Site 1 workers in the treatment group were more likely 
than those in the control group to say they planned to take a computer course, a home-study course, 
or courses leading to a two- or four-year college degree. Neither of the other sites showed any 
impacts in this area. Those in the treatment group at Site 1 were also more likely to report having 
signed up for another course of some type (not just NWLP courses) than were those in the control 
group-by a margin of 1 1 percentage points. Once again, this was the only site where such an impact 
occurred. 19 



18 For outcomes measured as “yes” and “no” we report the percent of workers in the treatment 
and control groups responding “yes” when computing impacts. Besides reporting impacts as the 
percentage-point difference, we provide effect estimates so these results can be compared with 
impacts measured in other scales. Differences of about 1 0 percentage points are considered small, 
differences of about 25 percentage points are considered medium-sized, and differences of about 40 
or more points are considered large (Cohen 1988). 

19 Many of the impacts we computed concerning future plans were negative; that is, members 
of the control group tended to be more likely to plan to take courses than were workers assigned to 
the courses. While most of the negative impacts were not statistically significant at conventional 
levels for a two-tailed test, the finding that controls were somewhat more likely than treatments to 
tell us that they planned to take a course may have resulted from workers in the control group being 
instructed that they could attend an NWLP course after the delay period ended. 
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TABLE III. 4 



IMPACT OF WORKPLACE LITERACY ON CAREER AND EDUCATIONAL PLANS, 
IN PERCENTAGE POINT DIFFERENCES (AND STANDARD DEVIATION UNITS) 



i Outcome 




Site 1 


f|U Site 2 




Site 3 


Changed educational or career goals 


11 


(.22)* 


-6 


(-.14) 


6 


(.16)* 


Planned to take: 


a basic skills course 


1 


(.02) 


-23 


(-.47) 


-11 


(-.23) 


a course in using English 


2 


(.07) 


-14 


(-.29) 


-17 


(-.46) 


a computer course 


9 


(.18)* 


-6 


(-.16) 


-11 


(-.26) 


a course to prepare for a GED exam 


4 


(.09) 


-5 


(-.29) 


-20 


(-.40) 


courses to get an occupational certificate 


-2 


(-.04) 


-6 


(-.13) 


5 


(.10) 


a job training course 


-6 


(-.13) 


-7 


(-.14) 


-12 


(-.24) 


courses leading to a 2- or 4-year college degree 


7 


(.23)* 


6 


(.14) 


11 


(.30) 


a home-study course 


10 


(.20)* 


-7 


(-.16) 


2 


(.04) 


Signed up for another course 


11 


(.26)* 


-11 


(-.29) 


-5 


(-.13) 



*Significant at the .10 level using a one-tailed test. 
NA = unable to estimate. 



c. Workplace Literacy Increased the Frequency with Which Learners Undertook Some 

Literacy-Related Activities at Home 

To assess whether workplace literacy affected participants’ literacy-related activities at home, 
we asked workers, how often they wrote various things (for example, checks, letters, and grocery 
lists) and how often they read various things (for example, letters or bills, street signs, and books). 20 
Our results indicate that workplace literacy can affect how often workers read a wide variety of 
materials at home, although it has little impact on the frequency with which workers write when at 
home. 

Of the 12 reading items we asked about, we found impacts in eight areas (see Table III. 5). The 
impacts range from about . 1 5 to .40 standard deviations, comparable in size to the impacts we found, 
for example, for self-reported ability to read and understand English. Furthermore, as with other 




20 In both cases, the response scale ranged from “not at all" to “regularly (every week).” 
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TABLE III. 5 



IMPACT OF WORKPLACE LITERACY ON HOW OFTEN WORKERS 
READ AND WRITE VARIOUS THINGS AT HOME, 

IN STANDARD DEVIATION UNITS 





Site 1 


|Q Site 2 f S 




How often workers read: 


letter or bills 


.16* 


-.07 


.02 


coupons 


.33* 


-.01 


-.25 


labels on food 


.37* 


-.05 


-.30 


food recipes 


.16* 


.02 


-.63 


religious materials 


.10 


-.24 


-.23 


instructions 


.19* 


.31* 


-.26 


street signs 


.30* 


.07 


.03 


newspapers 


.04 


-.04 


-.09 


notes from a teacher or school 


.39* 


-.17 


-.36 


TV Guide or other television listing 


.30* 


-.31 


.23 


magazines 


.04 


-.17 


-.06 


books 


.01 


-.05 


-.23 


How often workers write: 


checks 


-.01 


.16 


-.02 


notes or memos 


-.01 


.00 


.03 


food recipes 


-.11 


-.01 


-.44 


forms or applications 


.20* 


.40* 


-.45 


appointments on a calender 


-.04 


.28* 


-.43 


letters 


-.19 


-.01 


-.28 


stories or poems 


-.07 


-.15 


.01 


crossword puzzles 


.08 


.18 


-.11 


grocery lists 


.01 


.19 


-.17 


journal or diary 


.02 


-.27 


.00 



* Significant at the .10 level using a one-tailed test. 



outcomes described above, virtually all of the impacts on reading at home were concentrated in Site 
1 ; Site 2 showed an impact only in the frequency of reading instructions; Site 3 had no impacts. A 
somewhat interesting finding is that in these sites workplace literacy did not have an impact on 
reading behavior for materials of substantial length, such as newspapers, magazines and books, but 

ERIC 




did seem to change workers’ reading habits concerning typically shorter materials, such as coupons, 
food labels, recipes and street signs; items such as these are more typically part of the document- 
oriented reading curriculum in some courses. 

As for writing, we found impacts for only two of the 10 items we asked about, and these 
impacts occurred in only two of the three sites. In both Sites 1 and 2 workers selected for workplace 
literacy reported filling out forms or applications more frequently than their peers in the control 
group. And in Site 2 only we found an impact on how often workers write appointment on 
calenders. In all three cases, these impacts were in the small-to-moderate range. 



d. Workplace Literacy Changed Some Employment-Related Outcomes 

Our analyses of the impact of workplace literacy on job-related outcomes were based on 
workers’ self-reports about their recent employment experiences, employers’ reports on workers’ 
employment records, and supervisors’ ratings of workers’ performance on the job. When 
interpreting these findings, one must keep in mind that these are short-term impacts and some 
indicators concerning employment may not have had an opportunity to change during the 8- to 1 8- 
week course period. For example, promotion and pay raise decisions might only be made at one 
point in the year and will likely not coincide with the end of a site’s course cycle. 

Worker Reports. Our follow-up questionnaires asked workers about occurrences that had 
happened to them on the job since the time the course they took or signed up for had started: their 
employment status, their use of literacy skills on the job, their self-assessed teamwork skills, and the 
job benefits they were receiving. Although we found no impacts on employment status (for 
example, changes in employment or responsibility on the job), workplace literacy did have an impact 
on the last three; however, the results were inconsistent among sites (see Table III. 6). In both Sites 
1 and 2 workers in the treatment group rated their ability to work as part of a team as slightly higher 
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TABLE III. 6 



IMPACT OF WORKPLACE LITERACY ON EMPLOYMENT OUTCOMES, 
BASED ON WORKERS’ SELF-REPORTS, IN PERCENTAGE POINT DIFFERENCES 
(AND STANDARD DEVIATION UNITS) 



Outcome Site I Site 2 \ Site 3 



Since course started, worker: 



had more responsibility added to their job 


6 


(.12) 


-30 


(-.64) 


-2 


(-.05) 


moved to a preferred shift 


-5 


(-.12) 


-11 


(-.34) 


1 


(.03) 


switched from part-time to full-time work 


-13 


(-.37) 


-6 


(-.33) 


2 


(.07) 


received a pay raise 


-9 


(-.25) 


-11 


(-.37) 


-11 


(-.25) 


was promoted 


-9 


(-.31) 


-4 


(-.17) 


0 


(.00) 


received an award, bonus, or other special 












recognition on the job 


-6 


(-.18) 


-8 


(-.22) 


-3 


(-.07) 


applied for a new job 


-5 


(-.13) 


-4 


(-.11) 


1 


(.03) 


started a new job 


-4 


(-.12) 




NA 


1 


(.04) 


was laid off 


-4 


(-.24) 




NA 


-2 


(-.12) 


Employed (yes) 


-1 


(-.03) 




NA 


-7 


(-.17) 


Worker report of literacy skills used on the job: 














read instructions 


2 


(.04) 


-11 


(-•29) 


-3 


(-.06) 


receive spoken instructions in English 


7 


(.14) 


-1 


(-.03) 


-9 


(-.20) 


speak English 


11 


(.22)* 


1 


(.07) 


-11 


(-.23) 


work as part of a team 


-3 


(-.06) 


-6 


(-.30) 


15 


(.34)* 


write in English 


0 


(.00) 


-2 


(-.08) 


-1 


(-.02) 


use math 


-1 


(-.02) 


0 


(.00) 


11 


(.22) 


solve problems/use reasoning 


-4 


(-.08) 


-3 


(-.11) 


4 


(.08) 


Ability to work as part of a team 




(.31)* 




(.30)* 




(-.20) 


Benefits received on the job: 














paid vacation 


5 


(.10) 




NA 


0 


(.00) 


paid sick leave 


8 


(.17)* 




NA 


-2 


(-.04) 


paid holidays 


11 


(.23)* 




NA 


-7 


(-.15) 


health insurance 


9 


(.18)* 




NA 


-13 


(-.26) 



*Significant at the .10 level using a one-tailed test. 
NA = unable to estimate. 



(by about one-third of a standard deviation) than did their peers in the control group; participating 
workers in Site 3 reported they were more likely to work in teams. Workers in Site 1 also seemed 
to have benefitted from workplace literacy in terms of fringe benefits. Compared with the control 



group, the treatment group at Site 1 was more likely to have three of the benefits we asked about 
(paid holidays, paid sick leave, and health insurance); at Site 2 the impact was only on sick leave. 

While the impacts appear small-6 to 10 percentage points-the impact relative to the experience 
of nonparticipants is quite large. In Site 1 an impact of 8 percentage points on receiving paid sick 
leave corresponds to almost a 30-percent increase over what we would have expected if workers had 
not taken a course; for paid holidays and health insurance we found a relative increase of about 20 
percent. 

Although participating in workplace literacy had a large impact on the receipt of paid benefits 
such as sick leave and health insurance in Site 1, the results are puzzling. First, the program’s 
impacts on paid benefits occurred without the program affecting workers finding new jobs (see Table 
III. 6). Second, changes in paid benefits were occurring in a short period, while participation in the 
program affected none of the other outcomes concerning employment. What are some possible 
explanations for these findings concerning paid benefits? One possibility is that workers in the two 
groups changed jobs at the same rate, but those in the treatment group found employment in better 
jobs. Another possibility is that course participants negotiated better benefits with employers after 
learning about worker rights while attending courses. A third possibility is that after attending ESL 
courses, for example, workers are better able to understand what benefits are available to them at 
their jobs. Our data allow us to directly examine the first of these explanations-moving into jobs 
with different benefit packages-but not those concerning negotiation with employers and discovery 
of benefits already available. 

Only about 13 percent of the workers in Site 1 reported changing jobs between the time a course 
started and when the course ended. Among those who did change jobs, we find that workers who 
were in the treatment group were just as likely as those in the control group to have paid sick leave, 
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holidays, and health insurance (see Table III. 7). More specifically, the odds of receiving paid 
benefits are similar for both groups of workers. 21 A different picture emerges for workers who kept 
the same job; that is, members of the treatment group were much more likely to receive benefits than 
members of the control group. For example, among workers who did not change jobs, the chances 
of a worker in the treatment group receiving paid sick leave were 80 percent greater than for a 
worker in the control group (odds ratio is 1 .8). Given that workplace literacy had no impact on paid 
benefits among workers who changed jobs and did have an impact for those who kept their job, we 
must rule out the first explanation for our findings in Site 1 — that workers in the treatment group 
were getting better jobs than workers in the control group. We suspect that the two other 
explanations account for the short-term impact of workplace literacy on paid benefits, although 
which was more controlling remains unclear. 

Employer Reports. We asked employers in Sites 2 and 3 to report on seven measures of 
workers’ job performance, including whether they were still employed, their wages, and (concerning 
the preceding 1 2 months) whether they had been promoted, had received a bonus, the number of 
days late for work, number of days absent without an excuse, and number of days missed due to job- 
related accidents. For Site 3 we found that members of the treatment group missed about one less 
day per year because of job-related accidents than did workers in the control group (see Table III. 8). 




2l The odds of receiving paid sick leave, for example, are defined as the probability of receiving 
the benefit divided by the probability of not receiving the benefit. An odds ratio (the odds of 
workers in the treatment group receiving paid benefits divided by the odds for the workers in the 
control group) of 1 tells us that the chances of workers in the treatment group receiving paid benefits 
are the same as for workers in the control group. A value of less than 1 shows that members of the 
control group are more likely to receive benefits than the treatment group; a value greater than 1 
indicates that the treatment group is more likely to receive benefits than the controls. 
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TABLE III. 7 



ODDS OF RECEIVING PAID BENEFITS FOR WORKPLACE LITERACY 
PARTICIPANTS IN SITE 1 RELATIVE TO THE ODDS FOR 
MEMBERS OF THE CONTROL GROUP 3 





Paid Sick l^eave §§|. 


|!i! ? ipliday,: ; . 


Paid Health |||||| 
II 


Same job 


1.8* 


1.9* 


1.7* 


Changed job 


.7 


.8 


1.4 


♦Significant at the .10 level. 



“Odds ratios were computed by taking the results from a logit model that related receipt of a paid benefit to 
an indicator for treatment/control status and the propensity of being in the treatment group. Separate models 
were estimated for workers who changed jobs and for those who kept the same job. 



TABLE III. 8 

IMPACT OF WORKPLACE LITERACY ON EMPLOYMENT OUTCOMES, 
BASED ON EMPLOYER REPORTS, IN VARIOUS METRICS 
(INCLUDING STANDARD DEVIATION UNITS) 



, 1 1 Ififf-- §§§| |§§ |||| Illllil 


Site 2 


O' Site 3 | |(§ 


Employed (yes) a 




NA 


8 


(.16) 


Promoted in last 12 months (yes) a 




NA 


-6 


(-.12) 


Received performance-related bonus in last 12 months (yes) a 


-2 


(-.09) 


-5 


(-.11) 


Days absent without excuse in last 12 months 


.10 


(.26) 


.04 


(.05) 


Days late to work in last 12 months 


-.24 


(-.19) 


.33 


(.26) 


Days absent because of a job-related accident in last 12 months 


.20 


(.09) 


-.91 


(- 










.31)* 


Weekly wages b 


$17.71 


(.22) 


-$.62 


(.01) 



♦Significant at the .10 level using a one-tailed test. 

NA = unable to estimate. 

“Impact expressed as percentage point difference between treatment and control group. 
b For Site 2, the average weekly wage of study participants was $679; for Site 3 it was $278. 



BEST COPY AVAILABLE 




79 



103 



This was consistent with one of the objectives shared by employers in this site— improving safety on 
the job. However, we found no impacts on the other outcomes. 

Supervisor Ratings. For Sites 2 and 3 we asked supervisors to rate how well study participants 
performed selected tasks and demonstrated specific behaviors. 22 Results indicated that both sites 
showed modest impacts on how well workers performed certain tasks, but no impacts on how well 
workers demonstrated selected behaviors. 

At Site 2 supervisors rated workers assigned to the treatment group as performing better than 
their control-group counterparts on five of the tasks; at Site 3 this occurred for two of the tasks (see 
Table III. 9). Tasks on which workers in the treatment group outperformed those in the control group 
included arriving on time and ready for work, following supervisors’ verbal instructions, asking if 
coworkers or customers need help, answering simple job-related questions, giving clear 
directions/instructions to others, and completing tasks on time. Giving clear directions/instructions 
to others was the only task on which workplace literacy had an impact in both sites. The impacts 
ranged from about .3 to .4 standard deviations, quite similar in magnitude to those obtained for other 
outcomes reported above. These findings suggest that about 60 to 65 percent of the workers in the 
control group were given lower ratings by supervisors than the average worker in the treatment 
group. 



22 The rating scale had five points, ranging from “not well” to “very well. 



TABLE III. 9 



IMPACT OF WORKPLACE LITERACY ON JOB PERFORMANCE, BASED 
ON SUPERVISORS’ RATINGS, IN STANDARD DEVIATION UNITS 



Outcome • • V 1 


5* Site 2 


Site 3 


How well worker performs task: 


arrive on time, ready for work 


.40* 


.04 


follow supervisor’s verbal instructions 


.19 


.27* 


fill out simple forms accurately 


-.04 


.13 


write down a telephone message clearly 


.10 


.28 


ask if coworkers or customers need help 


.31* 


.19 


answer simple questions related to the job 


.34* 


.16 


give clear directions/instructions to others 


.33* 


.31* 


complete a task on time 


.40* 


.22 


follow safety rules on signs/in manuals 


.05 


.00 


understand memos/reports in English 


.02 


.02 


write memos/reports clearly in English 


-.04 


-.24 


solve problems using addition/subtraction 


-.05 


-.30 


solve problems using fractions/percentages 


.00 


-.27 


take measurements or make estimates 


.11 


-.06 


How well worker demonstrates behavior: 


give early notice of planned absences 


.21 


.07 


greet customers in a friendly manner 


.20 


.08 


cooperate well with coworkers on the job 


.04 


.01 


speak and contribute at worker meetings 


.09 


-.18 


tell supervisor about job-related problems 


.11 


-.01 


demonstrate a healthy sense of self-esteem 


.12 


.08 


complete work assignments enthusiastically 


.00 


.06 


pay careful attention to details of the job 


.05 


.16 


track down answers to complex questions 


.00 


-.07 


acquire new job-related skills quickly 


-.04 


.06 



* Significant at the .10 level using a one-tailed test. 



D. FACTORS ASSOCIATED WITH DIFFERENCES IN SITES’ EFFECTIVENESS 



The impact-study results presented above 
reveal that workplace literacy had impacts on 
many outcomes; however, the type and 
number of impacts varied dramatically 
between the sites. Site 1, as indicated in 
Table III. 10, showed program impacts on 
substantially more outcomes than Sites 2, 
which had many more impacts than Site 3. In 



FIGURE III.l 

PERCENTAGE OF OUTCOMES WITH 




Site 1 Site 2 Site 3 

fact. Site 1 had impacts on over 40 percent of 

the 62 “core” outcomes; 23 Site 2, in contrast, had impacts on a much smaller percentage of these 
outcomes; and Site 3 had impacts on almost none of these core outcomes (see Figure III.l). What 
might account for these differences in overall site effectiveness? 

One factor that may account for the differences in effectiveness is the amount of instruction that 
workers in the three sites received. At Site 1 class sessions were 3 hours long and courses lasted 
about 18 weeks; on average, workers attended about 77 percent of the class sessions and received 
an average of 43 hours of instruction through the first course they attended. Classes in Site 2 
typically met for 20 two-hour sessions spread over 10-13 weeks; participants attended, on average, 
84 percent of the class sessions, giving them an average of about 3 1 hours of instruction in their first 



23 For purposes of comparing the relative effects across sites, we included all outcomes from the 
follow-up questionnaire completed by the workers and results from the standardized literacy 
assessments. We did not include results from the supervisor form or the employer form because 
these data were not collected in Site 1 . It should be noted that Site 2 did achieve several impacts in 
areas assessed by supervisors. 
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TABLE III.10 



OVERVIEW OF SIGNIFICANT PROGRAM IMPACTS, BY SITE 



Outcome ||| ; i : - 


Site! §§ 


§p" ;: 


site 3-C 


Workers’ self-rated ability to: 








read English 


/ 


/ 




understand English 


/ 


/ 




speak English 


/ 






write English 


/ 






use math 


/ 






solve problems/use reasoning 


/ 






Standardized literacy assessment 


/ 






Changed educational or career goals 


/ 




/ 


Planned to take: 








a computer course 


/ 






courses leading to a 2- or 4-year college degree 


/ 






a home-study course 


/ 






Signed up for another course 


/ 






How often workers read (at home): 








letters or bills 


/ 






coupons 


/ 






labels on food 


/ 






food recipes 


/ 






instructions 


/ 


/ 




street signs 


/ 






notes from a teacher or school 


/ 






TV Guide or other television listing 


/ 






How often workers write (at home): 








forms or applications 


/ 


/ 




appointments on a calender 




/ 




Workers’ self-reported ability to work as part of a team 


/ 


/ 




Worker speaks English on the job 


/ 






Works as part of a team on the job 






/ 


Benefits received on the job: a 








paid sick leave 


/ 


b 




paid holidays 


/ 


b 




health insurance 


/ 


b 




Number of days missed work in last 12 months 








because of a job-related accident 


b 




/ 


Supervisor ratings of how well worker performs tasks: 


b 






arrive on time, ready for work 




/ 




follow supervisor’s verbal instructions 






/ 


ask if coworkers or customers need help 




/ 




answer simple questions related to the job 




/ 




give clear directions/instructions to others 




/ 


/ 


complete a task on time 




/ 





NOTH: All impacts shown were significant at the .10 level using a one-tailed test. 

a Any short-term impacts on these outcomes are produced by a program’s direct emphasis on benefits during instruction or counseling. 
b Unable to measure or to estimate effect for this outcome at this site. 
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course. At Site 3 classes typically met for 6 hours per week over a period of about 8 weeks; learners 
attended 68 percent of the class sessions and received, on average, only 19 hours of instruction. 24 

Disentangling the effect of instructional hours from other program components that might 
account for between-site differences was not easy; it required that we estimate an analytic model to 
assess whether, controlling for other factors, course participants with more hours of instruction 
scored higher or had better outcomes than participants with fewer hours of instruction. This analysis 
differed in three ways from the earlier analyses that compared workers in the treatment group with 
those in the control group: 



• Participant Sample. Only course participants (workers in the treatment group who 
attended at least one class session) were included in the analysis. 25 

• Aggregated Data. To maximize the variability in hours of instruction, we pooled the 
three sites into a single analysis. In doing so, we constructed a sample of workers with 
hours of instruction that ranged from 1 to 60 hours, with an average of 38; about 25 
percent of the participants received between 1 and 33 hours of instruction, 50 percent 



24 Adult education and workplace literacy experts generally believe that program operators 
should strive to provide learners with a substantial number of instructional hours in programs such 
as workplace literacy (Mikulecky and Lloyd 1993). However, programs like workplace literacy are 
often identified as giving learners relatively limited hours of instruction. For example, we previously 
reported that among NWLP learners nationwide who completed at least one course during an 1 8- 
month period, about half received fewer than 16 total hours of instruction (Moore, Myers, and Silva 
1997). 



“Overall, about 27 percent of the workers in the treatment groups at the three sites had zero 
hours of instruction, according to information the sites submitted to MPR through the NWLIS. We 
hesitate to make much of this, however, because we suspect for workers in some sites this may 
reflect a reporting error and not actual behavior concerning workers showing up for services. For 
example, during several course cycles we discovered that, in specific sites, there were many more 
workers with zero hours of instruction in the data base than during either the preceding or following 
course cycle. This pattern may result from changes in personnel entering information into the 
NWLIS or from poor record keeping by some instructors. 
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had between 34 and 48 hours, and the remaining 25 percent had more than 48 hours of 
instruction. 26 

• Outcomes Analyzed. The analysis was limited to only the 29 outcomes that we 
measured at two points— before each learner’s first course started and after that course 
ended. By focusing on outcomes measured at two points, we can better adjust for 
differences in personal characteristics among workers, which, in turn, allows us to more 
accurately estimate the net effect of hours of instruction on worker outcomes. 



It should also be noted that results from this analysis do not have the same power as those based 
on the randomized experiment, because selection effects still may bias our estimates of the 
relationship between hours of instruction and worker outcomes. Selection effects may exist because 
workers with many hours of instruction may be quite distinct, in ways we could not observe, from 
those with only a few hours of instruction. 27 Nonetheless, the findings may suggest the type and size 
of impacts that can be achieved when projects retain workers in courses for longer periods or when 
they link shorter courses together to achieve longer total exposure to instruction. 

The results showed that hours of instruction received by workers were related to 10 of the 29 
outcomes we analyzed. These outcomes are listed in Table III.l 1 along with estimates of the impact 
of one additional hour of instruction on each outcome. While these effects are small, so too is a one- 
hour increase in instructional time. To convey how much a more substantial increase in instructional 
hours might affect worker outcomes, in Table III.l 1 we also show the predicted effect of increasing 
workers’ instructional time by 20 hours. 



26 About 8 percent of the participants had more than 60 hours of instruction in their first course. 
Because this appeared to be more than they could obtain during a typical course, we excluded these 
workers from the analysis. 

27 Appendix G describes in detail the statistical approach we use to estimate the effect of hours 
on worker outcomes. The approach uses a procedure that under certain assumptions corrects for 
selection effects. 
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Considering the average hours of instruction received by workers in the three sites, 20 additional 
hours represent a huge increase. For Site 3 this would almost amount to the average worker taking 
a second workplace literacy course. In Sites 1 and 2, a 20-hour increase would represent roughly 
a 50-percent increase in the average hours of instruction workers receive. Putting aside the issue of 
what it might take to effect such a large increase in instructional hours, however, the results suggest 
that for the 10 outcomes listed in Table III. 1 1, substantial impacts may be achieved when hours of 
instruction are significantly increased. These results suggest that increasing learners’ hours of 
instruction can lead to changes in worker outcomes. Furthermore, these findings add support to our 
speculation that differences in sites’ effectiveness may be associated with differences in the average 
amount of instruction received by participating workers. 

Other factors, besides hours of instruction, also are likely to account for sites’ overall 
effectiveness. Drawing on observations from the case studies, we offer some possible explanations 
below for the differences in impacts we observed between sites. 



• Site Objectives. The national evaluation collected information on a broad range of 
potential outcomes aligned with the intent of the NWLP, but individual sites may have 
emphasized some objectives more than others. Site 1 had a primary aim of giving LEP 
workers sufficient English language skills to enable them to move to better jobs; Site 
2 emphasized instruction to improve workers’ math, writing, and team skills so they 
could better perform their current jobs; and Site 3 focused on upgrading workers’ basic 
skills, especially English language communication skills, and improving their behavior 
concerning job safety. 28 



28 To the extent that sites showed impacts on outcomes that corresponded to their particular 
program objectives, they might consider themselves effective despite not showing impacts on the 
core of measures assessed in this evaluation. 



TABLE III. 11 



CHANGES IN OUTCOMES AFFECTED BY INSTRUCTIONAL HOURS, 
IN STANDARD DEVIATION UNITS 



Outcome 


Increase in 1 
Instructional Time 


| -Iffeefbia 20^Hour | 
Increase in 

Instructional Time : 


Self-reported ability to use math 


.03 


.58 


Self-reported ability to solve problems 


.02 


.36 


How often worker: 


writes crossword puzzles 


.03 


.66 


reads letters/bills 


.02 


.36 


reads coupons 


.04 


.78 


reads labels on food 


.06 


1.10 


reads newspapers 


.02 


.46 


reads magazines 


.02 


.42 


Worker receives paid sick leave 


.03 


.68 


Worker uses math at the job 


.03 


.64 



Note: All results shown are significant at the .10 level using a one-tailed test. 



• Site Experience. When we started the impact study, the sites were at different 
developmental stages. Sites 2 and 3 were operating under their first NWLP grant, 
employed staff who were relatively new to workplace literacy, and were in the early 
stages of developing their workplace literacy curriculum. Site 1, however, was 
operating under its third NWLP grant, and employed instructors and other staff who had 
considerable experience with the workplace literacy model. 

• Course Scheduling. Sites scheduled courses for different times, which might have 
affected participants’ abilities to attend class sessions and master the material covered. 
Site 1 held teacher-led classes on Saturdays or Sundays, which did not interfere with 
learners’ work schedules; the site also offered child care to better enable parents to 
attend. Classes in Site 2 typically met during the workday, and the employer provided 
paid leave to participants. But in Site 3 classes were commonly scheduled immediately 
after the workday, when workers were often tired from their physically demanding jobs 
and had other demands on their time; this seemed far from optimal as a condition 
supportive of learning. 
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• Motivating Factors in the Workplace. Workers’ perceived prospects for getting better 
jobs differed among sites. Workers in Sites 2 and 3 expressed little optimism that 
promotional opportunities linked to improved literacy skills were available to them. 
Site 2 workers worried about the flatter organization they saw coming with a 
restructured plant, while Site 3 workers saw only limited opportunities to become 
supervisors. Site 1 workers, in contrast, were guided by the prospect of moving to better 
jobs in the industry as a whole and were not expecting major changes in their current 
work assignments. 



It is likely that differences in sites’ effectiveness are attributable to several of these factors. 
With only three sites in the impact study, we could not calculate the effects of such factors on 
program impacts; however, in Chapter IV we outline a framework to guide future investigations that 
could meet this research objective and build on the results of our exploratory evaluation. 

These findings reinforce the puzzle that workplace literacy programs present. Programs appear 
more likely to show positive gains in areas addressed directly by instruction (Mikulecky and Lloyd 
1993). Yet to focus on a broad number of skill areas through short duration courses risks 
accomplishing little to no impact. The difficult challenges for workplace literacy programs in the 
future center on obtaining sufficient instructional time and teaching skills that transfer beyond the 
narrow confines of workers’ immediate jobs so workers have more opportunities to develop those 
skills. 
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IV. SUMMING UP: CONCLUSIONS AND NEXT STEPS 



Findings in this report about the successful establishment of workplace literacy programs and 
employers’ decisions to continue such programs with their own funds are encouraging. That some 
workplace literacy programs improve workers’ skills, plans, at-home behaviors, and on-the-job 
performance is also encouraging. But findings in this report also indicate that some programs have 
very little impact on many of their participants. Thus, while offering encouragement to proponents 
who seek the expansion of workplace literacy to improve the literacy skills of adults in the labor 
force, the findings in this report leave a number of questions unanswered. Consequently, there is 
need for caution before answers from a larger sample of workplace literacy programs are available. 

A range of important questions have yet to be resolved. We don’t know, for example, how long 
employers will continue to support workplace literacy programs, nor do we know how these 
programs will retain a balance between basic literacy skills and job skills as they evolve. The 
infrastructure to foster effective workplace literacy programs remains in question in settings where 
there are few, if any, public funds to sustain it. We also don’t know how many workplace programs 
achieve impacts on many of the core outcomes examined in this study. And, if hours make a 
difference as our findings suggest, we must be concerned about the related finding that most 
workplace literacy programs give workers only modest amounts of instructional time. Finally, we 
lack evidence about the long-term effects of workplace literacy on workers-for example, increased 
wages, better jobs, and the acquisition of additional education. 

Definitive answers to the above questions were not the goal of this evaluation. Rather, this 
evaluation was undertaken to identify findings that future studies could validate in more sites and 
with more participants. Nevertheless, within this circumscribed focus, the evaluation has uncovered 
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important information about the challenges programs encounter in implementing and 
institutionalizing workplace literacy services and the implications of these challenges for future 
workplace literacy efforts. It also has documented short-term effects of three very different 
workplace literacy programs-effects that result from the programs and not the motivations of the 
individuals who enroll in them nor other changes in the workplace. 

In this chapter we summarize the conclusions that flow from findings of the national evaluation. 
Furthermore, we set forth a framework for future efforts to validate and expand upon these findings. 
The current study has demonstrated the feasibility of applying the elements of this framework. 
Nevertheless, we anticipate that this framework will be difficult to implement without centralized 
direction to ensure its faithful execution and to assemble comparable measures from a large number 
of sites across the nation. Support for such an undertaking is likely to fall primarily upon the public 
sector. It is clear that the major audience and future sponsors for validations of this study’s findings 
are public officials and employers who have not seen sufficient evidence to consider investing in 
workplace literacy. Because employers already investing in such efforts generally have limited need 
for exacting evidence to justify their programs, they are unlikely sponsors for the further validation 
defined in the framework. 

A. CONCLUSIONS 

We use the three main issues that guided this evaluation as challenges to the future of workplace 
literacy to frame our conclusions. 
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1. Is It Feasible To Establish and Institutionalize Workplace Literacy Programs? 

Overall, the evidence indicates that credible workplace literacy programs can be established in 
a variety of workplaces and it shows that workplace literacy programs with close ties to employers 
are likely to continue. More specifically, the evidence leads to the following conclusions: 



a. Establishing Local Programs Requires Time, Expertise, and Flexibility 

Policy makers and employers should expect that quality workplace literacy programs take time 
to develop and mature. While greater efficiency may be possible, these programs by and large will 
not fit into a turnkey strategy or top-down design. The local workplace literacy programs we 
observed adhered strongly to a grass-roots approach. Program staff spent time acquiring knowledge 
about the workplace and expertise in constructing and delivering a customized curriculum. Over 
time they were able to apply this knowledge and experience to new circumstances. This is an 
important reason why veteran programs and staff found it easier to resolve implementation issues. 
Even with experienced and knowledgeable staff, however, the development of programs entailed a 
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significant investment of time. 

Flexibility also was critical to the evolution of workplace literacy programs. Responding to 
changes in the workplace and heeding feedback from employees by redesigning courses, formats of 
courses (for example, individualized instruction instead of classes), and schedules were very 
important in programs’ ability to maintain the support of workers and managers. 

Two implications flow from these observations. First, evaluations that attempt to measure 
workplace literacy’s effectiveness should focus on programs that have had time to mature and are 
staffed by instructors with a year or more of experience. Second, the compelling goal of finding 
ways to make the process of program development more efficient will need to accommodate the 
underlying grass-roots, highly adaptive nature of workplace literacy . 
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b. Employer Support Is Critical for Continuation of Programs 

Workplace literacy programs tend to continue, at least in the short run, if they have acquired 
strong support from employers. All workplace programs in the in-depth study that were sustained 
after the NWLP demonstration grant had developed this kind of support. Support, was displayed in 
several ways, not all of which were essential for any one program. In this study, forms of support 
included employers’ commitment of appropriate space for a program, provision of full or partial 
release time, efforts to integrate skills taught in courses into workers’ jobs, and linking workplace 
training with other types of training. Further, internal company incentives such as international 
certifications of quality were important catalysts of employer support. Requirements for the gradual 
assumption of the full cost of the workplace literacy program also appeared to ease companies’ 
commitment of the financial resources for operating a program after public subsidies had ended. 

Reinforcement of our observation that programs’ survival depends on employer support was 
apparent in the continuation problems faced by workplace literacy programs with little direct access 
to employers. Public funds, philanthropic grants, or payments imposed on learners were the main 
survival alternatives for such programs, but these sources proved not to be very feasible. Based on 
our observations of programs that operate at a distance from employers, we speculate that publicly 
subsidized workplace literacy programs using this approach to serve workers from small businesses 
may be similarly vulnerable to continuation problems, unless there has been careful planning for 
long-term sources of support during implementation of the program. 

c. The Continuation of Workplace Literacy Programs for Workers in De-Skilled Jobs Will 

Require Sources of Support Other than Employers 

Workers with limited literacy skills who are employed in jobs that require a limited set of skills 
are not likely candidates for employer-financed workplace literacy programs. Once federal funding 
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was over, employers in this study who had adjusted the workplace to accommodate their employees’ 
limited English skills did not elect to fund a workplace literacy program on their own. The de- 
skilled character of these employers’ workplace was evident in workers’ reports that their jobs 

o 

required little in the way of reading instructions in English, writing in English, or communicating 
with supervisors in English. Yet the program in Site 1 which served workers who were currently 
employed in de-skilled jobs, was the most effective literacy program in the impact study, 
demonstrating that effective instructional programs can be established for this population of workers. 
Without recourse to public funds or other sources of support, however, the continuation of programs 
for workers in these jobs is unlikely. Not surprisingly, there appears to be an insufficiently 
compelling business reason for employers to underwrite instruction in literacy skills that are not 
critical to job performance. 1 

2. Is Workplace Literacy Effective? 

a. Short-Term Impacts Are Detectable Such as Oral Language, Teamwork, and Home and 
Individual Behavior 

The national evaluation’s findings suggest that linking literacy instruction with the workplace 
can improve, in the short-term, some aspects of employees’ job performance and more general 
literacy skills. Although many impacts we observed were concentrated in one site, the results from 
the evaluation suggest the kinds of outcomes that may be achieved on a broader scale. The short- 
term job-related impacts appeared to focus on certain aspects of employees’ behaviors that may help 
them better work in teams such as giving clear instructions, asking co-workers if they need help, and 



'Arguably, employers may be making short-sighted judgments. Safety on-the-job and reduced 
employee turnover may be linked in the long run to improved English proficiency. In the short run, 
however, workers’ access to supervisors and others who can translate for them bridges their English 
deficiencies. 
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completing tasks on time. Workplace literacy also may influence workers’ oral English proficiency, 
self-reported proficiency to read and write in English, to use math skills, and to solve problems. 
Furthermore, participating in workplace literacy courses can affect some literacy behaviors such as 
what workers read and write at home, and on their educational and career goals. Taken as a whole, 
the results provide evidence that workplace literacy can lead to changes in many worker outcomes; 
however, given the large differences in effectiveness of the sites, the results also suggest that other 
factors may influence sites’ overall effectiveness. 

b. Various Factors, Including Instructional Time, Affect Why Some Programs Have More 

Impacts than Others on Core Outcomes 

A number of factors could account for the substantial variation we observed in sites’ 
effectiveness. Sites differed along a variety of dimensions including course emphasis, amount of 
instructional time received by participants, experience of the instructional staff, curriculum 
development, and course scheduling. Because the national evaluation used only a few sites for the 
impact study, disentangling the effects of these and other factors bn each site’s effectiveness is 
impossible. However, two pieces of evidence point to instructional time as a potentially important 
element. First, the most effective site across a broad range of outcomes had the highest average for 
hours of instruction received by participating workers. The site with the second highest average for 
instructional hours had some impacts, and the site with the fewest hours of instruction had almost 
no impacts. Second, even after taking into account other factors that could be related to 
improvements in literacy skills, more hours of instruction received by individual workers often were 
associated with substantial gains in participants’ outcomes. 
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c. Substantially Increasing Instructional Hours for Participants Would Require a Major 

Redirection of Current Practices in Workplace Literacy Programs 

If additional research confirms findings in this report and ties program outcomes to employees’ 
accumulation of 35 to 50 hours of instruction, it will confront program staff and employers with a 
tough challenge. Hours of this magnitude are not typical for the majority of workers who participate 
in workplace literacy programs, nor are these hours typical for workers who participate in more 
general job training. Sixteen hours of instruction is the national norm in for workplace literacy 
programs and formal job training (Moore, Myers, and Silva 1997). Organizational pressures and 
employees’ personal concerns are significant impediments to increasing the instructional time 
available in the workplace. In fact, a tendency among several continuing workplace programs is to 
reduce instructional time and the frequency of course meetings to minimize program cost, intrusions 
on shifts, and problems in employees’ course attendance. 

So far, the record of experience with alternatives that substantially increase instructional time 
is not very promising. Since few workers currently enroll in two or more courses, this path to 
increased instructional hours appears difficult to effectuate. Learning time can be increased by take- 
home instructional videos, and exploiting opportunities for practice during or surrounding the 
workday. Ultimately, however, these techniques may only result in marginal increases in time. It 
may be that programs could extend their educational counseling to acquaint workers with 
educational programs at technical schools or colleges and to find methods to ease their participation 
in these programs. While assistance to help workers pursue additional learning is part of many local 
programs, it often is delivered in a low-key manner when workers request such guidance. More 
active promotion of educational opportunities may be desirable if the objective is to increase the time 



workers devote to learning. 



d. Employers Evidence about Workplace Literacy’s Effects on Productivity Is Imprecise 
and Inseparable from Other Organizational Reforms 

Most employers already involved in workplace literacy programs seek internally credible 
information about programs effects on their own workplaces but they do not seek exacting measures 
of programs’ contributions to productivity. As noted in the second chapter of this report, anecdotal 
comments from supervisors and employees that reflect their own observations often suffice as 
evidence of how programs aid productivity. While statistical data that attest to changes associated 
with the introduction of workplace literacy programs may exist, these data are not often available to 
those beyond the ranks of upper management. Moreover, even if they were available, these data 
usually do not disentangle whether the workplace literacy programs are the source of improvements 
in company performance on indicators such as the frequency of safety violations, employee 
attendance records, rework orders, and reduced employee turnover, or whether other organizational 
factors are responsible. As a general rule, such distinctions were not of concern to employers in this 
study. 

This observation has two implications. First, employers with established workplace literacy 
programs are not a force for obtaining more precise information than what they already can summon 
within their own organizations. In short, employers sponsoring workplace literacy programs are 
unlikely to play a major role in proposing additional research to demonstrate workplace literacy’s 
effects on productivity. Second, to clearly identify the contribution of workplace literacy to employer 
productivity will require research that involves the careful collection of information within companies 
over a period of time, ideally starting before or at the point the company initiates a workplace literacy 
programs. Such research will need to document various changes and reforms taking place within the 
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workplace which may or may not be linked to workplace literacy, and find ways to separate the 
contributions of these factors and workplace literacy to measures of company performance. 

3. What Infrastructure Is Necessary to Support Workplace Literacy Programs? 

Infrastructure is essential if a system of workplace literacy is to replace episodic, stand-alone 
projects. Building the infrastructure for such a system requires time and sustained support. While 
this study focused on state and local infrastructure housed in a system of community or technical 
colleges, we expect that the many of the issues we uncovered are common to infrastructure that is 
housed in other types of education providers . 2 

a. Infrastructure at the State and Local Levels Aids in the Establishment of Quality 

Workplace Literacy Programs 

The benefits of state and local infrastructure were evident from several perspectives. In terms 
of the institutionalization of programs, both state-level partnerships that were the focus of our 
examination of infrastructure achieved an impressive record of continuations. Institutionalization is 
in many respects a product of local programs’ successful navigation of implementation issues. Local 
implementation issues were more easily resolved by sites that had access to the shared experience and 
technical support that infrastructure provided. For example, instructors were critical to managing and 
promoting an on-going program of workplace literacy, but most required on-the-job training in unique 
aspects of the workplace and the instructors’ role in the workplace. Networks formed by a cadre of 
state leaders and local college mentors provided a systematic way for instructors to acquire necessary 
skills, and to draw upon techniques and curricula found to be successful in other environments. 



2 This study did not focus on federally subsidized infrastructure. Nevertheless, it should be noted 
that national conferences, newsletters, and research sponsored by the U.S. Department of Education 
have been components of a national infrastructure for workplace literacy. 



Since a number of states are just contemplating a workplace literacy infrastructure or are in early 
stages of its development, questions of strategy are particularly important. Based on our 
investigation, we expect that strategies across the states will emphasize different components of 
infrastructure. While this study cannot point to the most effective components, it suggests the 
importance of developing components that are sufficiently flexible to accommodate instructors in 
diverse work environments. Core curricula, for example, may be a beneficial component of 
infrastructure, but demonstrating to instructors that the core affords the flexibility required in a variety 
of local contexts may be a major challenge. Similarly, an infrastructure based on process has to 
contain elements of flexibility. For example, we found that the process elements defining Site 5’s 
model of workplace literacy-that is, governance structures, policies on employees’ voluntary 
participation, and individualized instruction— had to allow for workplaces in which some of these 
procedures were incompatible. Decentralized organization also is an important feature of 
infrastructure because it is conducive to local flexibility. 

b. The Continuation of Infrastructure at State and Local Levels May Be Uncertain Without 
Public Support 

The prospects for continuation of an infrastructure for workplace literacy differ somewhat for 
state and local organizations. Local infrastructure housed in community colleges, for example, 
arguably could be sustained by a local market for workplace literacy— that is, a continuing demand 
from employers for external assistance in establishing and operating workplace literacy programs. 
To be a sustaining source, however, the market needs to produce a stable flow of assistance contracts 
or purchase orders, which may be difficult to realize. Economic reverses in a region, for example, 
may limit the ability of employers to finance the start-up costs for a workplace literacy effort. 
Unstable demand can lead to the loss of experienced staff and the decline in support services provided 
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by local colleges, thus eroding local infrastructure. While at this time no one can forecast whether 
market forces alone will sustain local infrastructure, we observed two states in which the degree of 
certainty surrounding continuation of local infrastructure was quite different. It is clear that in the 
state with less uncertainty, state supplied start-up grants made a major difference because they 
secured the positions of staff dedicated to building and maintaining workplace literacy programs at 
colleges around the state. 

There is little question that maintaining infrastructure at the state level depends heavily on public 
sources of support. Employers’ and employees’ demands can be a key factor in stimulating the 
commitment of public funds to support state level activities and staff directed at developing 
workplace literacy capacity, but there is no market for employer purchased services as there is at the 
local level. 

B. A FRAMEWORK FOR EXTENDING THE FINDINGS 

Several lessons emerge from the current study about how to structure future efforts to extend the 
findings in this report. 

1. Importance of the Value-Added Approach 

A key feature of any future evaluation of short-term impacts should be the use of a value-added 
design consisting of the random selection of workers into workplace literacy services and a control 
group. Without such a design, isolating the impact of workplace literacy from the influence of the 
many other changes taking place simultaneously in the work environment is very difficult . 3 Although 

3 Uncovering the long-term effects of workplace literacy will probably have to rely on the 
statistical analysis of longitudinal data. While this approach will not produce findings with the same 
level of credibility as the value-added design advocated in this report, it is probably the only practical 
way of learning about the effects of workplace literacy on outcomes such as wages, employment, 

(continued...) 



previous evaluations of workplace literacy have not used this approach, the current study 
demonstrated the feasibility of using this design for computing short-term impacts. Furthermore, we 
can show that when a less rigorous design is used, we would reach substantially different conclusions 
about the effectiveness of workplace literacy. 

To illustrate the consequences of using a less rigorous design, we focused on a set of items asking 
workers to report their ability to understand, speak, read, and write in English, to use math, and to 
solve problems. Also, to compare designs, we used the kind of information that many workplace 
literacy sites may already collect; that is, information about workers’ literacy skills measured before 
they started a course and then measured again at the end of the course (a pre/post design). 

Among the sites, we found that the random selection and pre/post designs produced noticeably 
different conclusions. For Site 1 we found that the two designs produced similar results for almost 
three-quarters of the items; however, a very different picture developed in Sites 2 and 3. In both sites, 
the two designs produced inconsistent results for the majority of the items. For Site 2 the pre/post 
design suggested that workplace literacy was often ineffective when random selection showed that 
impacts were present and in Site 3 the pre/post comparison of outcomes suggested that workplace 
literacy often was effective when random selection showed no impact. If we relied on the pre/post 
design, we might conclude that the services offered in Site 2 generally had no impact and those in Site 
3 were very effective, this is just the opposite conclusion we reached when we compared the 
outcomes for the randomly selected treatment and control groups. Differences in results obtained 
from the two designs may stem from workers’ literacy skills changing independently of their 



3 (... continued) 

and job promotion. To learn about such outcomes with a random selection design would require 
employers to hold members of the control group out of workplace literacy for long periods, 
something that most employers would find onerous. 
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participation in workplace literacy courses. Such changes could occur, for example, because some 
participating workers were seeing their literacy skills improve as a result of on-the-job training, and 
not workplace literacy. 

2. Public Policy Interest and Strong Incentives Are Necessary to Accomplish the Expanded 

Evaluation Effort 

When we negotiated sites’ participation in the impact study, employers and education providers 
often told us that participating would be inconvenient or disruptive to the workplace. Furthermore, 
sites had little incentive to participate because from their perspective the data we were collecting did 
not align well with their own evaluation strategy nor would the results generally affect their decisions 
concerning the long-term future of the program. Although employers in this study found few reasons 
to participate in the study or to demonstrate a program’s effectiveness empirically, elected officials 
have an obligation to report statistically credible evidence concerning program impacts when 
justifying the commitment of public funds to an activity. 

Implementing rigorous evaluations that encompass a number of local workplace literacy 
programs will be challenging. There must be a centralized authority that provides direction to ensure 
the framework is faithfully implemented and strong incentives for workplace literacy sites to 
participate in such an evaluation. Since we would expect that public support will be used to fund 
future demonstrations, linking the receipt of a grant to participation in the evaluation would provide 
a powerful incentive. Without such an incentive, it is likely that the next evaluation will encounter 
the same obstacles faced in the current study and few sites will be willing to come forward. 

3. Data Collection Should Not Burden Programs 

Each impact study site had trouble collecting data from workers, supervisors, and employers. 
Even the simplest data collection tasks, except for recording class attendance, burdens the staff and 



can disrupt their normal activities. Because of these difficulties we would not expect sites to collect 



data in future evaluations; instead, the primary 
responsibility for collecting data should be given to 
professional data collectors from outside the program. 4 
This should ensure that high quality data are collected 
and that site operations will run smoothly. 

4. A Meta-Analysis Model for Combining Results 

The framework we have outlined to this point calls 
for well-designed studies that will produce credible 
results concerning the impacts observed in individual 
workplace literacy sites. To extend our findings and to 
learn about the approaches that are most effective under 
different circumstances calls for methods that permit 
combining findings from diverse sites. Statistical 
procedures for conducting such analyses-methods for 
meta-analyses-are already well-developed and are 
routinely used in a variety of contexts (see, for example, 
Hedges and Olkin 1985; Bryk and Raudenbush 1992). 
The basic elements of a meta-analysis include measures 
of program impacts for a set of core outcomes and a list 



Core Outcomes for Measuring 
Short-Term Impacts 

Data Collected from Workers 

• self-reported literacy skills 

• career and educational plans 

• literacy behaviors at home and work 

• literacy skills used on the job 

Data Collected from Supervisors 

• ratings of how well worker performs 
tasks 

• ratings of how well worker 
demonstrates job related behaviors 



Factors Affecting Program Effectiveness 
Worker Characteristics 

• hours of instruction 

• worker demographics 

Characteristics of the Workplace 
Program Characteristics 

• scheduling of courses 

• course emphasis and scope 

• teacher’s experience instructing adult 
learners 

• instructional approaches 

• availability of support services and 
counseling 

Program Costs 



4 This would represent an extension of the approach used in the current study where MPR 
focused on collecting follow-up information from members of the control group, and asked the sites 
to collect data from all workers before courses started and from members of the treatment group after 
courses ended. 
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of factors believed to influence each site’s effectiveness. Once these data are assembled, statistical 
analyses can be conducted to show the influence of various program factors on each site’s impact. 

Although we asked each of the impact study sites to use a standardized literacy assessment, we 
have not included it as one of the proposed core outcomes for two reasons. First, unless programs 
are experienced using standardized literacy assessments, a strong possibility exists that program staff 
will inappropriately administer the tests. For example, before giving workers a standardized literacy 
assessment, staff at one site in the impact study failed to administer the publisher’s locator test to 
assess what level of test they should give workers. Consequently, many workers may have taken a 
literacy test that was either too difficult or too easy. Second, and perhaps most importantly, only 
when a standardized literacy assessment is regarded as highly consistent with a site’s curriculum, is 
it likely that the program will produce changes in workers’ test scores. Site 1, which had a large 
impact on a test of oral English proficiency, appeared to have a close fit between its program 
objectives and the literacy test, and used the assessment forjudging its own success independent of 
our evaluation. Sites 2 and 3, where we found no impact on test scores, did not see the literacy 
assessment they adopted for the national evaluation as central to their assessment of learners’ 
progress. Instead, they viewed it as necessary for ED’s evaluation of workplace literacy and not of 
particular relevance for the partnership. While standardized literacy tests can be useful tools for 
assessing program impacts, we suspect that we will learn little from their application unless sites are 
well versed in their use and the tests are considered as central to the endeavor. 

The evaluation framework we have proposed for validating and extending the findings in the 
current evaluation will give public policy makers credible information about the short-term 
effectiveness of the workplace literacy model. The current study showed that while the framework 
is feasible, implementing it will be challenging. It calls for both centralized direction to ensure its 
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faithful implementation, strong incentives so that workplace literacy sites will participate in an 
evaluation, and an appropriate set of common measures to allow combining results across a wide 
range of contents and curricula. 
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Site 1 



Amount of NWLP Support: 1 5643,735 

Three partners comprise the workplace literacy program at Site 1 : a nonprofit organization that 
provides a range of literacy programs and vocational training primarily to Asian immigrants in the 
city where it is located, a textile workers union, and an association representing several hundred 
garment manufacturers. The literacy- and vocational-training provider serves as the education 
partner. Founded in 1972, the training provider has received various grant funds from the U.S. 
Department of Education and the city’s employment agency since 1973; the 1994 NWLP grant was 
organization’s third in a row. The other two partners have helped develop the workplace curriculum 
and assisted with recruiting workers for the program. In addition, the union partner has begun using 
portions of the curriculum developed under the NWLP grant in some of its own ESL courses. The 
program has very little contact with employers of the learners. 

The project offers three ESL courses and three basic skills courses. The basic skills courses are 
for the more advanced leamers--those with greater oral and written English proficiency. Typically, 
courses meet on Saturdays and Sundays for about 1 8 weeks, with each class session lasting three 
hours. Participation is entirely voluntary; employers often do not know that their workers are 
attending classes, and do not offer bonuses or paid release for attendance. 

The cornerstone of the program is formal in-class instruction, complemented by the use of 
computer-assisted instruction. Instructors use several approaches in the classroom, including 
instructor-led presentations, role playing, and hands-on activities. Class instruction is highly 
structured and prescribed. Besides using several modes of instruction, over time the program has 
developed written, audio, and, more recently, video materials. The videos are used in the classroom, 
can be checked out by students who miss classes, and are also broadcast on a local television station 
that caters to Asian viewers. The television broadcasts are available to participants and 
nonparticipants (workers not officially enrolled in program courses). Nonparticipants can even 
obtain homework assignments and self-administered tests that the program scores for them. 

In addition to ESL and basic skills courses, the program provides a variety of other services to 
participating learners, including child care, job and family counseling, and referral services. The 
core program staff consists of the project director, a full-time curriculum developer, three part-time 
instructors, and a part-time counselor. The project also uses several volunteers as tutors. 

The program served about 270 learners during the first 1 8 months of our study. More than 90 
percent of the learners were women and almost all were foreign bom. One-fourth of the learners 
have completed less than eight years of schooling, almost 30 percent have 9-1 1 years of schooling, 
and only about 15 percent have attended school for 12 or more years (for 30 percent of the learners 
we could not determine their years of schooling). More than 80 percent of the learners were 
employed and, on average, they had worked at their current jobs for three years. When asked what 




'Represents three years of funds from the 1994 cycle awards, 
nonfederal funds to meet or exceed partnership match of 30 percent. 
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activities they did at their jobs, 60 percent or more of the workers responded that they did not read 
instructions, receive instructions in English, speak English, write in English, or use math. 

Learners averaged about 66 hours of instruction by the end of the first 1 8 months of our study. 
About three-fourths of the learners had taken one course and the remaining one-fourth had taken two 
or more courses. Almost 40 percent had completed at least one course. 



Site 2 

Amount of NWLP Support: 5900,000 (est.) 

The workplace literacy project at Site 2 is a collaboration among five partners: a metal 
processing plant that employs about 600 people, an agency that supplies temporary workers to the 
plant, a community college, a local organization promoting literacy, and the county government's 
adult education program. The program operates on the premises of the metal processing plant, and 
is designed to help the company’s employees master the literacy skills required in a technology- 
based work environment. The company’s interest in a workplace literacy program was prompted 
by a variety of factors, including the adoption of Statistical Process Control to monitor and control 
processes throughout the plant, an older workforce in need of retraining, and the corporate 
commitment to maintain ISO 9002 certification. 

Program operations have been guided by two committees. A steering committee, comprised of 
representatives from the collaborating entities and two production workers, has played an informal 
role in reviewing the program’s activities and recruiting participants. An education advisors team, 
with front-line worker representatives from each department, crew, and shift, has been instrumental 
in advising on program policy, program purchases, recruitment, and the content and schedule of 
classes. 

The program’s curriculum consists of five core courses: Foundation Skills, Communication for 
the 90’s, Team Survival Skills, Applied Math, and Learning to Learn. These courses address the two 
areas of skills required by the plant’s workers: (1) functional literacy skills, including math, reading, 
writing, and leaming-to-leam skills, and (2) participatory team skills, including communication, 
problem-solving, and decision-making. Each course is based on a listing of core tasks that workers 
are expected to perform and incorporates materials and examples from four areas of the plant- 
potline, cast house, carbon plant, and maintenance. In addition to attending classes, program 
participants are able to use a learning lab to supplement their skill development with computer-based 
instruction and also have access to a resource library that contains over 200 books, audio tapes, video 
tapes, and other educational materials. Workers may also receive help from tutors provided by the 
literacy association and may attend the adult high school diploma program and GED preparation 
classes offered by the county adult education program. 

Courses follow two schedules. Day classes last two hours each and meet 20 times over a 10- 
week period. Night shift classes are also two hours long, but the 20 sessions are spread over a 13- 
week period. Participants are assessed with a pre- and post-test designed specifically for each 
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course, and receive a certificate of completion if they achieve 80 percent on the post-test or a gain 
of 1 0 percent between pre- and post-test if the pre-test score was 80 percent. 

According to anecdotal reports, workers have increased their ability to communicate effectively 
with co-workers, participate in team meetings, and solve production problems in the workplace. 
These assessments of the program’s success have convinced the company to continue operating the 
program for at least one year after the end of the federal grant. The program next year is likely to 
institute shorter courses (perhaps totaling four instructional hours) that are delivered in a module or 
workshop format and that focus on specific skills related to team functioning and other critical areas 
in a technology-oriented workplace. 



Site 3 

Amount of NWLP Support: $1, 171, 606 

The focus of the workplace literacy program at Site 3 is upgrading the basic skills— especially 
English language communication skills — of workers in agriculture and related industries. The 
participating entities are a community-based nonprofit organization that has traditionally focused 
on job training and community development; five businesses, including meat, fruit, and vegetable 
processing companies; a union; and a business that provides assistance and training to small minority 
businesses on topics such as promoting entrepreneurism, creating jobs, and management training. 
The last organization was only briefly active in the partnership. The key factors that promoted the 
formation of this partnership were the agriculture industry’s need to retain workers and increase the 
safety conditions in plants, as well as provide opportunities for workers to advance themselves. 

Advisory committees were established in the early months of the program to assist with the 
identification of the content for instruction and the recruitment of workers, but these activities were 
eventually undertaken by the program’s instructors in conjunction with each company’s project 
contact person at the work site. The instructors work with company staff in organizing the 
instruction and in recruiting participants by including flyers in paychecks, displaying posters, and 
distributing program information at new employees’ orientation. 

The program has provided two levels of ESL instruction and GED preparation for employees 
at their work sites. Through the Essential and Intermediate English courses, workers learn skills in 
English speaking, listening, reading, and writing and apply these skills to tasks in the workplace. 
Applications have included reading charts, graphs, and other materials, communicating with 
supervisors and co-workers, and completing workplace forms. Classes are held for one and a half 
hours, four days a week; the courses typically last eight weeks. Workers also are given assignments 
to complete at home to enhance their learning, but workers report these have been difficult to find 
time to do. 

A competency-based curriculum has been designed for the program that includes 42 
competencies in the two levels of English skills. Lesson plans and worksheets have been developed 
for use in instruction. Participants are assessed with the CASAS pre- and post-test, the adult 
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Language Assessment Scale (LAS-0), and applied performance assessments that were developed in 
collaboration with the national evaluation to measure the skills taught in the curriculum. 

Workers receive a certificate of completion after they have finished a course and attended at 
least 20 hours of instruction. Events celebrating participants’ completion of the course also are held 
at the work sites. 

Information collected by the partnership through follow-up interviews with 24 participants and 
anecdotes provided by company personnel suggest that participants have improved their skills and 
made some advances in the workplace. One company representative also reported that worker 
turnover appeared to have decreased, although data had not been analyzed to document this outcome. 



Site 4 



Amount of NWLP Support: $733,831 



The program at Site 4 was first funded under the NWLP in 1 99 1 , received a second federal grant 
for 1992-1994, and a third grant in January 1995. The goal of the third grant has been to prepare 
employees to work in high performance workplaces focusing on total quality management (TQM). 
The state community college board is the grantee, and administers the project through its office of 
adult literacy. As part of the third grant, workplace literacy programs were begun with a total of 10 
employers. Formal partnerships were formed with six employers (three manufacturing firms, one 
hotel, one medical center, and a mortgage company) and five education service providers (four 
community colleges and one community-based organization). In addition, the project established 
workplace literacy programs with four other employers who were not formally designated as partners 
in the NWLP grant (a clothing company, two hotels, and a manufacturer of surgical instruments). 
Each instructional program has been based at the work site, and focused on developing the core basic 
skills that enable employees to perform successfully in challenging, high performance workplaces. 
These skills have included reading technical documents, writing, applied math, making 
presentations, higher-order thinking, and decision-making. Several of the work sites adopted a 
TQM-related curriculum that emphasized ESL instruction. 



The state-level partnership has used a decentralized approach in which the state program 
coordinator has facilitated the activities of local partnerships by providing technical assistance, 
convening lead instructors to share their experiences, disseminating information, and serving as an 
ex-officio member of the local task forces that were formed at each of the work sites. The lead 
instructor from the education service provider (usually a community college) has coordinated the 
services offered at the work sites and has been the main contact with the employers. The task force 
at each site has included representatives from management, as well as employees in the program. 
The task forces have carried out a variety of activities, including identifying instructional needs, 
reviewing curricula, developing instructional materials using workplace realia, and marketing the 
program. An important initial goal of the grant was to develop a curriculum template that specified 
the core skills needed for TQM that could subsequently be customized to different employers and 
industries. Due to concerns held by local lead instructors, the partnership decided that the sites 
would first develop their own curricula instead of developing the template and local curricula 
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simultaneously. These local curricula are now being incorporated into a template as a final grant 
activity. 

Various formats have been used for delivering instruction in the project work sites, including 
formal courses of several weeks duration, short courses, tutoring, and project-based learning using 
teams. Key to the program’s successful establishment of site-based services has been instructors* 
capacity to respond to changing demands in the workplace, and to be flexible in designing courses 
and delivering instruction. The instructors have come to be viewed as important staff within the 
work sites, which has contributed to their ability to customize the instruction. 

A significant focus in the program, from the perspective of the manufacturing employers, has 
been upgrading employees’ skills related to ISO 9000 certification and documenting these skill 
improvements. In addition to the basic skills that workers need to perform their jobs, the ability of 
workers to engage in lifelong learning and to communicate effectively in the workplace also have 
seen as critical by some employers. Although the program did not collect formal assessment data, 
informal data gathered through observations and anecdotes suggested managers saw the programs 
as effective in developing employees’ skills. 

When the grant ends, seven of the employers will continue offering some form of workplace 
literacy instruction. Furthermore, the instructional program at one employer is being pilot tested in 
two of the corporation’s other locations. At the state level, the community college board is searching 
for funds to continue workplace literacy services as a state initiative. 



Site 5 

Amount of NWLP Support: $2, 617, 838 

The workplace literacy program at Site 5 has existed as a statewide initiative for 10 years. 
During those years the program has been supported by five federal grants and variable amounts of 
annual funding from the state’s technical college system. The succession of federal grants has 
assembled a stable team of state partners who deliver guidance, information, and technical assistance 
to local workplace partnership projects throughout the state. Since the time of the 1994 federal 
grant, the partnership has consisted of the state technical college board, a university research center, 
the state branch of a major labor union, and a statewide association of employers. The state 
employers’ association, however, has not played a major role in the partnership. The technical 
college board, the union, and the state employers’ association have been partners throughout the 
succession of federal grants. 




The current federal grant has supported 20 local workplace education partnerships across the 
state during the past three years. Each local partnership has established a learning center that is 
located at or close to employees’ workplaces. Each local partnership has included one of the state’s 
technical colleges and an employer. Employers in the current grant and previous grants have come 
almost exclusively from the manufacturing sector, reflecting a statewide priority to retain 
manufacturing as a basic component of the state’s economy. It was this goal that led the state to 
initiate a statewide workplace education program with state funds from the technical college system. 
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The program builds upon a network of technical college districts across the state. Over two- 
thirds of the colleges (11 of 1 6) formed partnerships with employers under the recent federal grant. 
Coordinators from the colleges supply the know-how necessary to help employers analyze their 
educational and training needs, to identify the basic skills requirements of jobs, to hire instructors, 
and to establish a functioning learning center within the company. During the grant, day-to-day 
operation of the learning centers, recruitment of workers, design of curriculum, delivery of 
instruction, and feedback to the workers and employers was the responsibility of the college-hired 
instructor(s) at each center. Instructors’ ability to reach out within the companies and effectively 
market the center’s resources have been critical requirements of staff in these positions. 

The state partnership has defined a model of workplace literacy for local partnerships that 
includes: education at the work site; a gradually increasing cash contribution from employers; 
voluntary, open entry/open exit instruction; an emphasis on individualized, self-paced instruction; 
confidentiality related to workers’ involvement in the center; thorough analysis of workplace needs 
and the education skills required in jobs; and local site-based steering committees and groups of peer 
advisors. The model has served as a guide; local employers’ preferences have necessitated flexible 
use of the model’s elements. 

The state’s approach to building local workplace literacy programs has involved considerable 
technical assistance. The technical assistance occurs through a variety of statewide meetings, 
publications containing guidance and ideas for all aspects of developing a learning center, training 
sessions, and in-person as well as telephone consultations for local partnerships. The state 
partnership has developed a number of guides describing activities and approaches that instructors 
and coordinators from the local partnerships consider effective. The guides also outline steps for 
conducting job skills analyses and for training peer advisors. Technical assistance activities are 
covered primarily by the federal grants the state has received. The workplace literacy state grants 
program does not support a similar degree of networking and technical assistance. 

Fifteen of the 20 employers that began a learning center under the 1 994 federal grant have made 
plans to support centers on their own in the coming year. A major issue for employers has been 
whether to continue the program in concert with the technical colleges. Three-quarters of the 
continuing employers have seen the educational resources of the colleges as worth the extra expense 
of college-employed instructors. Several instructors, coordinators, and state partners also expressed 
concern that confidentiality for workers would erode in the absence of the colleges’ participation. 
There also was some concern that the centers would lose a basic skills perspective and be pushed by 
internal company pressures toward job-training. 
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The three local and two state-level partnerships that are the focus of this in-depth study were 
chosen to highlight a range of approaches in delivering workplace literacy instruction. This diversity 
is borne out in comparisons of the sites on several key dimensions, including the characteristics of 
the learners they served, the amount and type of instruction the learners received, the strategies used 
to provide instruction, and the activities of key players in the partnerships. We highlight these 
differences in this appendix, with references to national estimates of the NWLP from our Interim 
Report (Moore, Myers, and Silva 1 997).’ 



A. PARTICIPANTS’ BACKGROUND CHARACTERISTICS 

Nationwide, NWLP learners were an average of about 38 years old, about half were women, 
slightly more than half (55 percent) were white, one quarter were foreign bom (immigrants), nearly 
a third (30 percent) were limited English proficient (LEP), 2 and two-thirds had completed 12 or more 
years of schooling. (Table B.l displays data on each of these background characteristics for all 
participants nationally, along with data for participants in each of the five in-depth study sites.) 

The personal background characteristics of the workers served by each of the in-depth study 
sites varied substantially. For example, in Site 1 nearly all the participants were female, whereas in 
Site 2 women accounted for less than one-tenth of the learners; a large majority of the learners 
served in Sites 1 and 3 were LEP, compared with much lower proportions at Sites 2 and 5; and in 
Site 4 about two-thirds of the learners had at least 12 years of schooling in the United States and/or 
abroad, compared with higher proportions in Sites 2 and 5 and lower proportions in Sites 1 and 3. 

From the information presented in Table B.l, a fairly clear picture emerges of the type of 
learners involved at each site, revealing varying degrees of homogeneity, with some sites much more 
diverse than others. Site 1 almost exclusively served female Asian/Pacific Islander immigrants, most 
of whom had limited English language skills and less than 12 years of formal education. In contrast, 
Site 2 almost exclusively served American-born males, about equally divided between blacks and 
whites, and nearly all of whom had at least a high school education. Site 3 served primarily foreign- 
bom Hispanics with limited English language skills, most of whom had less than 12 years of formal 
education. In Site 4, the participants were mainly women, two-thirds had completed 1 2 or more 
years of schooling, almost half were bom outside the United States, about one-third were 



'Integrated with the text are six tables, typically showing percentages of learners, courses, etc., 
with certain characteristics. At the end of the appendix are six corresponding tables, showing the 
number of cases (Ns) on which the percentages were calculated. Data presented in this appendix, 
from the 1 8-month NWLIS reporting period, may not match data presented elsewhere, especially 
pertaining to Site experiences during the impact study, because (1) the random assignment period 
extended beyond the NWLIS period, and (2) in some cases, NWLIS data may have been reported 
erroneously. 




2 We defined learners as LEP if they reported their ability to speak or understand English was 
only “poor” or “fair.” 
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TABLE B.l 



PARTICIPANTS’ BACKGROUND CHARACTERISTICS 





Nationiil 


y$iie 1 | 








WM 


Age 

—Mean 


37.53 


39.4 


41.04 


33.91 


37.26 


38.60 


—Median 


37 


5 


40 


33 


37 


38 


—Youngest 


16 


40 


20 


17 


18 


18 


—Oldest 


79 


21 


63 


64 


64 


76 






63 










Percent Female 


49.8 


92.2 


6.5 


44.2 


64.7 


41.0 


Race/Ethnicity (%) 
—White, Non-Hispanic 


55.1 


0 


45.3 


10.5 


55.9 


91.5 


—Black, Non-Hispanic 


16.5 


0 


50.2 


0 


7.1 


4.3 


—Hispanic (All Races) 


17.7 


0 


2.5 


88.0 


2.9 


2.0 


—Asian/Pacific Islander 


8.2 


100 


0.8 


0.4 


33.5 


1.1 


—American Indian/Alaskan Native 


2.4 


0 


1.2 


1.1 


0.6 


1.1 


Percent Foreign-Bom 


25.2 


99.6 


3.3 


83.3 


44.8 


2.1 


Percent Limited English Proficient 
(LEP) a 


30.0 


98.9 


22.8 


79.3 


37.1 


9.0 


Prior Educational Attainment (%) 
—No Schooling 


0.8 


0.4 


0 


7.2 


1.2 


0.1 


-1-5 Years 


3.8 


10.0 


0 


22.1 


4.7 


0.4 


-6-8 Years 


6.8 


16.4 


1.6 


24.6 


3.5 


0.9 


-9-11 Years 


17.5 


29.7 


9.5 


18.1 


12.8 


9.2 


-12+ Years 


67.3 


13.8 


88.1 


17.8 


66.9 


89.0 


—Undetermined 


3.7 


29.7 


0.8 


10.1 


11.0 


0.5 



a We classified learners as LEP if they rated their own ability to speak or understand English as 
“poor” or “fair.” 
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Asian/Pacific Islanders, and a slightly higher proportion were LEP. Finally, the “typical” learner in 
Site 5 was white, American bom, and high school educated, and the majority were men. 



B. PARTICIPANTS’ EMPLOYMENT-RELATED CHARACTERISTICS 

Nationally, nearly all NWLP participants were employed when they first enrolled in a workplace 
literacy course, they had worked at their jobs for an average of just over seven years, and almost one- 
fifth held more than one job. 3 For an indication of job quality, we asked which of four fringe 
benefits— paid sick leave, health insurance, paid vacations, and paid holidays— learners received; 
more than half received all four benefits and just less than one-third received three of four. Learners 
typically worked 40 hours per week or more, and slightly less than one-fourth were members of a 
union. Finally, large majorities of learners reported that their jobs required them to do each of seven 
activities we asked about: read instructions, receive instructions spoken in English, speak English, 
write in English, use math, work as part of a team, and solve problems/use reasoning. (Table B.2 
displays data on each of these job-related characteristics for all participants nationwide, along with 
data for participants in each of the five in-depth study sites.) 

As with personal background characteristics, the job-related characteristics of learners served 
varied considerably between sites. For example, in Site 2 almost none of the learners received all 
four types of benefits we asked about (because their employer did not provide paid sick leave), but 
about 91 percent received the other three benefits, whereas in Site 4 about 83 percent received all 
four; only about 41 percent of Site 1 participants usually worked 40 hours per week, compared with 
about 75 percent in Site 5 and about 93 percent in Site 3; and the percentage of learners who were 
members of a labor union varied from only about 1 percent at Site 2 to about 61 percent at Site 1 . 



C. LEARNERS’ COURSE TAKING, COMPLETION RATES, AND HOURS OF 
INSTRUCTION 

About 75 percent of all NWLP participants nationwide enrolled in only one course during the 
18-month data collection period, about 14 percent enrolled in two courses, and about 10 percent took 
three or more. About 48 percent of all learners completed at least one course during the same time 
period. For a variety of reasons, however, this rate should not be interpreted as an indicator of 
student or program success. 4 Finally, among those participants who completed one or more courses, 



3 A11 descriptive information on learners’ jobs pertains to the job that allowed them to take a 
workplace literacy course. 

4 These factors include (1) the type of courses offered; (2) when classes finished-learners would 
not have had a chance to complete courses still under way when the data collection period ended; 
and (3) how project officials defined “completion.” 
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TABLE B.2 



PARTICIPANTS’ EMPLOYMENT STATUS AND WORK-RELATED CHARACTERISTICS 





National 

Estimate 


Site 1 




Site if 


-IfiteJ: 


Site 5 | 


Employment Status (%) 














—Employed 


96.6 


84.4 


100 


96.8 


99.4 


99.3 


—Temporary Layoff 


0.4 


3.0 


0 


1.1 


0.6 


0.1 


—Retired 


0.1 


0 


0 


0 


0 


0.2 


—Not Employed 


3.0 


12.6 


0 


2.2 


0 


0.4 


Percent with More Than One Job 


18.8 


0.4 


35.3 


8.6 


24.1 


21.2 


Benefits Received on Job (%) 














—Paid Sick Leave 


64.2 


26.9 


1.7 


53.4 


85.2 


51.9 


—Health Insurance 


90.2 


52.8 


96.6 


79.6 


95.9 


96.3 


—Paid Vacations 


92.2 


35.5 


99.2 


78.4 


93.6 


95.4 


—Paid Holidays 


93.1 


63.0 


93.7 


76.5 


89.3 


97.8 


—All 4 Benefits Above 


60.1 


18.3 


1.7 


49.8 


82.6 


50.9 


—3 of 4 Benefits Above 


29.7 


14.0 


90.8 


25.7 


2.3 


43.3 


Mean Number of Years Worked at 


7.3 


3.3 


12.3 


4.2 


4.3 


9.3 


Current Job 














Hours Worked Per Week (%) 














—More Than 40 


19.3 


19.6 


41.5 


5.6 


4.1 


22.9 


-40 


71.6 


40.9 


53.5 


92.9 


80.8 


75.2 


-Less Than 40 


9.1 


39.6 


5.0 


1.5 


15.1 


1.9 


Percent Union Members 


23. 2 a 


61.3 


1.2 


40.8 


15.7 


38.8 


Activities Need To Do on Job (%) 


b 












—Read Instructions 


85.6 


36.8 


90.0 


43.9 


89.2 


93.3 


—Receive Instructions Spoken in 














English 


88.6 


36.8 


92.1 


68.0 


95.8 


92.0 


—Speak English 


92.5 


39.1 


97.9 


59.6 


100 


97.7 


—Write in English 


82.7 


12.8 


92.9 


38.4 


86.5 


96.0 


—Use Math 


78.1 


33.3 


90.0 


39.8 


74.1 


93.1 


—Work as Part of aTeam 


93.0 


58.7 


97.5 


68.8 


100 


94.1 


—Solve Problems/Use Reasoning 

Q D rk ^ m 1 1 — * * * A. *J.1_ 1 _ a _ /\ T 


86.8 


48.1 


91.6 


52.3 


96.4 


95.8 



a Based on all participants with data (N=l 9,687); figure in Interim Report (24%) was based on only 
participants who were employed or on temporary layoff (N= 18,969). 
b Data not included in Interim Report. 
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the mean hours of instruction received was 30; however, the median was just 16. 5 (Table B.3 
presents data on course taking, completion rates, and hours of instruction for all participants 
nationwide, along with data for each of the five in-depth study sites.) 

Learners’ experiences at the in-depth study sites differed from the national picture and from one 
another in various ways. 

• Number of Courses Taken. The percentage of learners who enrolled in only one course 
during the data collection period ranged from about 62 percent in Site 3 to about 100 
percent in Site 5. Over 25 percent of learners in Sites 1 and 3 enrolled in two courses, 
and in Site 3 an additional 1 1 percent took three or more classes. 

• Course Completion Rates. The percentage of participants who had completed one or 
more courses when the data collection period ended varied greatly between sites, 
ranging from about 5 percent in Site 5 to about 69 percent in Site 2. It should be noted, 
however, that the self-directed learning model followed in Site 5 probably made it 
difficult for officials there to determine whether/when participants had completed a 
course of study. Also, learners in Site 1 who missed more than one class out of 1 8 were 
not counted as having completed the course— a more rigorous definition than was used 
elsewhere. In Site 3 learners were considered to have completed a course if they 
attended at least 20 hours. Therefore, as noted above, a site with a relatively high course 
completion rate should not be viewed as more successful than a site where the course 
completion rate was a relatively low. 

• Total Hours of Instruction Received. The mean hours of instruction that participants 
received ranged from about 22 in Site 4 to about 66 in Site 1. 6 We cannot provide a 
good estimate for Site 5, because the learning center approach used there meant that 
very few learners were counted as “completers.” 



D. COURSE EMPHASIS, ENROLLMENT LEVELS, TIMING, INSTRUCTION, AND 

LEARNER ASSESSMENT 

What types of courses were offered as part of the NWLP nationwide, and how were they run? 
The most common type of course offered focused on basic skills/literacy (31 percent), followed by 
ESL (20 percent). Courses typically met during the workday (but not during lunch) or immediately 
before or after normal working hours. Regardless of the major emphasis, a given course can teach 
a variety of skills; nationwide, more than half of all courses reportedly taught problem solving and 
communication, while math was very seldom taught (apart from reading). More than half of all 
courses frequently or always used team learning, teacher-led classrooms, and workplace 
documents/displays; in contrast, fewer than half used audio visuals or self-paced learning to that 



5 Data on hours of instruction were generally available only for course completers 

6 In all five in-depth study sites, the median hours of instruction was fairly close to the mean. 
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TABLE B.3 



PARTICIPANTS’ COURSE ENROLLMENT, ATTENDANCE, AND COMPLETION 





■il Estimate 


\-Site 1. 


Site 2 


1 Site 3^ 


r$ite4:V 


SiteS.:- 


Number of Courses Taken (%) 
—One 


75.4 


74.0 


79.9 


61.9 


86.2 


99.9 


—Two 


14.4 


25.7 


18.1 


27.0 


12.6 


0.1 


—Three or More 


10.2 


0.4 


2.0 


11.2 


1.1 


0 


Percent Completed at Least One 
Course 


47.7 


36.8 


68.9 


45.7 


35.6 


5.2 


Total Hours of Instruction 

Received 3 

-Mean 


30.1 


66.2 


39.5 


27.3 


22.3 


n.a. 


—Median 


16 


62.5 


36 


19 


19 




—Fewest 


2 


56.5 


1 


1 


4 




—Most 


245 


101 


98 


82 


47 




Enrolled in Course with Various 
Instructional Methods Used 
“Frequently” or "Always” (%) 
—Team Learning 


b 

63.5 


100 


36.2 


79.6 


86.2 


0 


-Teacher-Led Classroom 


53.6 


100 


9.1 


62.6 


25.2 


19.7 


-Computer-Assisted Learning 


29.2 


0 


76.0 


0 


5.0 


86.2 


—Self-Paced Learning 


43.3 


0 


81.1 


21.5 


36.5 


77.2 


—Audio Visuals 


56.2 


100 


14.4 


50.0 


88.7 


40.9 


—Workplace Documents/Displays 


58.3 


100 


34.2 


74.1 


40.9 


26.1 


Enrolled in Course with Various 
Primary Emphases (%) c 
—Basic Skills/Literacy d 


b 

28.5 


0 


47.1 


26.9 


0 


36.1 


-ESL 


14.0 


48.5 


0 


46.4 


23.2 


0 


—Speaking, Listening, 
Communication 


11.6 


51.5 


11.0 


9.0 


18.9 


0 


-Writing Skills 


3.9 


0 


11.6 


1.2 


0 


0 


—Problem Solving 


6.5 


0 


3.9 


0 


13.0 


0 


— GED Preparation 


3.6 


0 


5.5 


3.5 


0 


0 


—Team Building 


4.1 


0 


0 


0 


7.0 


0 


—Motivational 


7.4 


0 


0 


12.7 


33.5 


0 


—Other Unspecified 


20.5 


0 


21.0 


0.2 


4.3 


63.9 
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TABLE B.3 (continued) 





P’I^(t|pjaaI 

: Estate 


' Site |f; 




§f Site 3:'. •' 


Site 4 


.Site. -5: 


Enrolled in Course with Various 
Skills Taught (%) 

—Problem Solving 


b 

68.0 


100 


88.2 


78.2 


52.8 


49.0 


—Communication 


65.4 


100 


91.3 


88.6 


69.2 


55.0 


—Writing 


51.3 


100 


47.2 


91.5 


46.5 


79.1 


—Motivational 


48.4 


31.6 


22.0 


61.3 


57.9 


43.6 


—Team Building 


49.4 


0 


81.1 


58.3 


57.9 


36.7 


—Reading and Math 


41.8 


0 


71.3 


12.9 


17.6 


88.9 


—Reading 


23.8 


100 


22.4 


87.5 


24.5 


12.0 


-Math 


8.8 


0 


6.3 


14.0 


0 


12.0 


-Other 


22.0 


100 


0 


3.7 


52.2 


57.1 


Enrolled in Course that Met 
Various Times of Day/Week (%) e 
—During Workday (not at lunch) 


b 

70.2 


0 


100 


45.4 


50.9 


55.7 


—Immediately Before/ After 
Workday 


43.8 


0 


16.1 


100 


36.5 


54.8 


—At Lunch 


5.4 


0 


13.0 


2.6 


0 


33.1 


—On Weekends 


2.6 


100 


13.0 


0 


0 


0 


—Other 


7.3 


0 


7.9 


0.4 


19.5 


23.3 



“Calculations are based only on learners who completed at least one course and who had greater than 
0 instructional hours. For the national estimate, published in the Interim Report, we also first 
trimmed the distribution to eliminate several extreme values. However, the absence of such data for 



the five in-depth study sites made this analytical step unnecessary. 
b Information was not reported in Interim Report. 
c Based on duplicated count of learner-courses. 

includes “Basic Skills-Reading Only,” “Basic Skills-Math Only,” “Basic Skills-Reading and 
Math,” and “Literacy/Pre-Literacy.” 

Percentages do not sum to 100 because courses learners took could have met various days/times, 
n.a. = not available. 
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extent, and less than one-fifth frequently or always used computer-assisted learning. 7 Finally, the 
single most common assessment method used for placing or evaluating learners was interviews, but 
even more courses used one or more of several “other” methods. (Table B.4 presents data on all 
these course characteristics for the NWLP overall, and for each of the five in-depth study sites.) 

As with other dimensions of workplace literacy instruction, course schedules varied between 
the in-depth study sites. For example, all the courses at Site 2 met during the workday (although not 
during lunch), nearly all the courses at Site 3 met after the workday, and none of the classes at Site 
1 met at either of these times. 



E. STAFF DEVELOPMENT AND INVOLVEMENT BY EMPLOYERS AND UNIONS 

Nationwide, instructors have a variety of staff development opportunities. On one end of the 
scale, at about 87 percent of partnership programs were instructors involved in curriculum 
development/lesson planning; on the other end of the scale, only at about 15 percent of partnership 
programs were instructors involved in employer recruitment. Nationally, employers and unions 
involved in partnership programs rarely provided transportation or child care to learners; such 
services generally appeared unnecessary since courses were typically scheduled at or near the work 
site and either during or immediately before or after working hours. However, about 3 1 percent of 
employers/unions provided partial paid release— and an almost equal percentage provided complete 
paid release— to enable workers to participate in courses, and a majority provided participants with 
some form of recognition upon completion of a course. (Table B.5 includes data on these 
dimensions of workplace literacy projects for all projects in the country combined, and separately 
for each of the five in-depth study sites.) 

The in-depth study sites varied in the number and type of staff development opportunities 
offered to their instructors, and in the way employers/unions were involved. 

• Staff Development At Site 3 instructors reportedly were given the opportunity to 
engage in 9 of the 1 0 types of activities we asked about. In contrast, only three types 
of staff development opportunities were available to instructors at Site 1, and only four 
types were used at each of the other three sites. Curriculum development/lesson 
planning was the only activity available to instructors at every site; Site 3 was the only 
one to allow instructors to recruit employers; and none of the sites provided instructors 
with the opportunity to teach supervisors how to provide on-the-job reinforcement. 



’Practices that were not used “always” or “frequently” could have been used to a lesser extent 
(“sometimes”). Through our site visits we discovered that computer-oriented learning was under- 
reported by sites due to ED’s concerns that NWLP funds be used only to support the basic literacy 
skills associated with computer use. 
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TABLE B.4 



COURSE CHARACTERISTICS 



National . ’• 

Estimate lllilili. Site 2 Site ^ Site 4 ;I Site 5 



Instructional Methods Used 
Overall 3 (% Used “Frequently” or 
“Always”) 

—Team Learning 59 

-Teacher-Led Classroom 52 

-Computer-Assisted Learning 1 7 

-Self-Paced Learning 39 

—Audio Visuals 43 

—Workplace Documents/Displays 57 

Primary Emphasis (%) 

—Basic Skills/Literacy b 31 

--ESL 20 

—Speaking, Listening, 

Communication 1 1 

—Writing Skills 8 

—Problem Solving 7 

— GED Preparation 6 

—Team Building 3 

—Motivational 2 

—Other Unspecified 14 

Skills Taught (%) 

—Problem Solving 62 

—Communication 55 

—Writing 46 

—Motivational 40 

—Team Building 38 

—Reading and Math 27 

—Reading 23 

-Math 13 

-Other 1 5 

Percent With Curriculum 

Uniquely Developed for Course 69. 6 C 



100 


35.3 


57.1 


76.5 


0 


100 


11.8 


78.6 


23.5 


16.7 


0 


58.8 


0 


5.9 


88.9 


0 


64.7 


42.9 


23.5 


88.9 


100 


6.3 


46.4 


70.6 


33.3 


100 


35.3 


42.9 


50.0 


23.5 


0 


41.2 


22.2 


0 


44.4 


50 


0 


40.7 


29.4 


0 


50 


5.9 


11.1 


17.6 


0 


0 


5.9 


3.7 


0 


0 


0 


11.8 


0 


11.8 


0 


0 


17.6 


14.8 


0 


0 


0 


0 


0 


17.6 


0 


0 


0 


0 


17.6 


0 


0 


17.6 


3.7 


5.9 


55.6 


100 


82.4 


55.2 


64.7 


55.0 


100 


70.6 


58.6 


70.6 


65.0 


100 


52.9 


79.3 


58.8 


70.0 


27.8 


11.8 


37.9 


58.8 


45.0 


0 


52.9 


27.6 


58.8 


40.0 


0 


58.8 


20.7 


29.4 


90.0 


100 


17.6 


69.0 


23.5 


10.0 


0 


5.9 


17.2 


0 


10.0 


100 


0 


3.4 


47.1 


65.0 


100 


58.8 


75.9 


100 


11.1 
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TABLE B.4 (continued) 





National 

• : : :;;4§$timate: : ; 


Site 




Site 3 : 


Site 4 


Site 5 


Assessment Methods Used to 
Place or Evaluate Learners (%) 
—Interviews 


32 


100 


82.4 


55.2 


58.8 


85.0 


—Individual Learning Plan 


30 


0 


41.2 


0 


70.6 


60.0 


—Customized, Job-Related 
Competency Assessment 


26 


50.0 


41.2 


6.9 


47.1 


40.0 


—Standardized Literacy Test 


26 


50.0 


88.2 


34.5 


0 


55.0 


—Portfolio Assessment 


17 


0 


11.8 


6.9 


0 


25.0 


— Attitude/Self-Esteem Inventory 


14 


0 


11.8 


0 


0 


5.0 


—Supervisor Ratings 


14 


0 


0 


6.9 


35.3 


15.0 


—Learners’ Work Examples 


13 


0 


0 


24.1 


0 


25.0 


—Other 


37 


100 


47.1 


58.6 


35.3 


55.0 


Times of Day/Week Course Met 
(%) d 

—During Workday (not at lunch) 


58.3 


0 


100 


31.0 


47.1 


55.0 


-Immediately Before/After 
Workday 


42.3 


0 


17.6 


93.1 


23.5 


60.0 


—At Lunch 


2.9 


0 


5.9 


3.4 


0 


35.0 


—On Weekends 


2.4 


100 


5.9 


0 


0 


0 


—Other 

aT„* — : n a . i . 


8.8 


0 


5.9 


3.4 


29.4 


30 



“Interim Report also gave separate data for “classes” and “learning centers, workshops, and tutorials.” 
b Includes “Basic Skills-Reading Only,” “Basic Skills-Math Only,” “Basic Skills-Reading and 
Math,” and “Literacy/Pre-Literacy.” 



'Information was not reported in Interim Report. 

Percentages do not sum to 1 00 because a given course could meet at more than one time of the 
day/week. 



14 



O 




B-10 



TABLE B.5 



STAFF DEVELOPMENT AND INVOLVEMENT BY EMPLOYERS AND UNIONS 
(REGARDLESS OF PARTNERSHIP STATUS) 





|$atiOTiaJ 
llislimlte” i 


Site 1 


■ Site 2 


” Site3' : ! 


Site 4 


Site 5p 


Types of Staff Development 
Offered to Instructors (%, y/n) 
—Curriculum Development/ 
Lesson Planning 


87 


Yes 


Yes 


Yes 


Yes 


Yes 


—Teaching in the Workplace 


82 


No 


Yes 


Yes 


Yes 


Yes 


—Learner Assessment 


80 


No 


No 


Yes 


Yes 


No 


—Teaching Adults 


69 


No 


Yes 


Yes 


No 


No 


—Using Computer-Assisted 
Instruction 


57 


Yes 


Yes 


Yes 


No 


No 


—Learner Recruitment 


57 


No 


No 


Yes 


Yes 


Yes 


—Teaching ESL 


38 


Yes 


No 


Yes 


No 


No 


—Learner Counseling 


38 


No 


No 


Yes 


No 


Yes 


—Teaching Supervisors to Provide 
On-the-Job Reinforcement 


30 


No 


No 


No 


No 


No 


—Employer Recruitment 


15 


No 


No 


Yes 


No 


No 


Percent of Employers/Unions 
Provided or Paid for 
Transportation 


4 


100 


0 


0 


0 


6.5 


Percent of Employers/Unions 
Provided or Paid for Child Care 


1 


100' 


0 


0 


0 


6.5 


Percent of Employers/Unions 
Provided Release Time or Other 
Incentives/Recognition 
—Partial Paid Release 


31 


0 


0 


0 


14.3 


74.2 


—Complete Paid Release 


28 


0 


100 


0 


85.7 


0 


—Award Certificate at Completion 


53 


0 


100 


66.7 


85.7 


58.1 


—Award Ceremony at Completion 


36 


0 


0 


66.7 


14.3 


35.5 


—Cash Bonus Upon Completion 


6 


0 


0 


0 


0 


6.5 



“Percent of projects that offered. 
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. Employer/Union Involvement. Employers and unions involved in the three local and 
two statewide projects typically did not provide transportation or child care to learners. 
Site 1 was an exception with the union providing both types of services to enable 
learners to attend classes held on weekends. The union involved in the Site 1 
partnership was also an exception in that it did not provide any of the types of incentives 
or recognition for learners that we asked about, while the employers/unions involved 
with the other in-depth study sites typically provided at least a certificate to course 
completers. Such certificates and other forms of recognition were provided by the 
education provider in many programs nationwide. The measures presented here do not 
reflect this source of recognition. 



F. INSTRUCTORS’ CHARACTERISTICS AND EXPERIENCE 

Table B.6 provides information on the personal background characteristics and prior experience 
of NWLP instructors nationwide and at each of the five in-depth study sites. An interesting use of 
these data is to compare the instructors in a given site with the learners enrolled at the same site. For 
example, all instructors at Site 1 were Asian/Pacific Islanders and female, a profile similar to that 
of the site’s learners. In contrast, while Hispanics constituted 88 percent of learners in Site 3, only 
one of four instructors employed there on whom we had racial/ethnic data was Hispanic. And 
although blacks accounted for half of the learners at Site 2, all of the instructors there were white. 
Such demographic differences between instructors and learners were common nationally, as well. 
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TABLE B. 6 



INSTRUCTORS’ CHARACTERISTICS 



National 





Estimate 2 ^ 


| SiteT'l 






||Site 4 


jSit^ ; 5j 


Percent Female 


75 


33.3 


100 


25.0 


75.0 


69.6 


Race/Ethnicity (%) 


-White, Non-Hispanic 


83 


0 


100 


75.0 


100 


91.3 


-Black, Non-Hispanic 


6 


0 


0 


0 


0 


4.3 


—Hispanic 


3 


0 


0 


25.0 


0 


4.3 


-Asian/Pacific Islander 


5 


100 


0 


0 


0 


0 


—American Indian/Alaskan Native 


1 


0 


0 


0 


0 


0 


-Other 


1 


0 


0 


0 


0 


0 


Prior Educational Attainment (%) 


-High School Only 


<1 


0 


0 


0 


0 


0 


-Some College, but No Degree 


3 


0 


0 


0 


0 


0 


-Two-Year College Degree 


1 


0 


0 


0 


0 


0 


-Four-Year College Degree 


23 


33.3 


0 


80.0 


0 


13.0 


—Some Graduate-Level Credits, 
but No Advanced Degree 


19 


0 


0 


0 


25.0 


26.1 


-Master’s Degree 


45 


66.7 


100 


20.0 


75.0 


60.9 


-Ph.D. 


3 


0 


0 


0 


0 


0 


Prior Experience (%) b 


-Working in Industry/Sector 


56 


100 


60.0 


20.0 


100 


83.3 


—Teaching Secondary School 


40 


33.3 


60.0 


80.0 


50.0 


70.8 


—Teaching College 


38 


33.3 


80.0 


20.0 


50.0 


95.8 


—Teaching ESL 


38 


100 


60.0 


80.0 


75.0 


37.5 


—Teaching in the Workplace 


64 


0 


40.0 


80.0 


100 


95.8 


—Teaching Basic Skills Other than 
ESL to Adults 


59 


0 


40.0 


80.0 


25.0 


95.8 


Percent with ESL Training 


39 


100 


60.0 


80.0 


50.0 


45.8 


Percent Bilingual 


21 


100 


20.0 


60.0 


25.0 


20.8 


Percent with State Teaching 
Certificate 


44 


33.3 


60.0 


0 


50.0 


95.8 



“Rather than describing all NWLP instructors across the country as one group, the data from the 
Interim Report (reproduced here) described the average characteristics of instructors in projects, that 
is, project percentages, rather than raw numbers, were averaged. 

b Percentages do not sum to 100 because instructors could have experience in more than one area. 
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BEST COPY AVAILABLE 



G. DATA TABLES SUPPORTING TABLES B.l - B.6 



TABLE B. la 

NUMBER OF CASES USED IN CALCULATING DATA FOR TABLE B.l 
(PARTICIPANTS’ BACKGROUND CHARACTERISTICS) 





liitiorial 
Estimate | 




Site 2 


' Site 3 J 


Site 4 ' 


' Site'S;-- 


Age 


19,765 


269 


243 


265 


164 


1,581 


Percent Female 


21,043 


269 


248 


278 


173 


1,636 


Race/Ethnicity 


20,282 


269 


243 


276 


170 


1,629 


Percent Foreign-Bom 


20,026 


269 


244 


275 


172 


1,636 


Percent Limited English Proficient 
(LEP) 


18,521 


269 


246 


275 


170 


1,360 


Prior Educational Attainment 


20,024 


269 


243 


276 


172 


1,613 
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TABLE B.2a 



NUMBER OF CASES USED IN CALCULATING DATA FOR TABLE B.2 
(PARTICIPANTS’ EMPLOYMENT STATUS AND WORK-RELATED 

CHARACTERISTICS) 





||||{ationaLr; 


Y-Sitj! | 


Site 2'| 


SlteA 


f SiteAiil 


Site 5 


Employment Status 


20,603 


269 


246 


277 


173 


1,635 


Percent with More Than One Job 


17,449 


234 


238 


266 


170 


1,398 


Benefits Received on Job 


—Paid Sick Leave 


17,795 


234 


231 


266 


169 


1,396 


—Health Insurance 


17,963 


235 


238 


269 


170 


1,396 


—Paid Vacations 


18,049 


234 


236 


269 


172 


1,398 


—Paid Holidays 


18,017 


235 


238 


268 


168 


1,395 


—Number of Benefits 


17,643 


235 


239 


269 


172 


1,398 


Years Worked at Job 


18,327 


234 


240 


252 


171 


1,296 


Hours Worked Per Week 


18,607 


235 


241 


267 


172 


1,395 


Percent Union Members 


19,687 


269 


241 


277 


166 


1,426 


Activities Need To Do on Job 


—Read Instructions 


18,126 


234 


239 


269 


167 


1,414 


—Receive Instructions 


18,085 


234 


241 


269 


166 


1,416 


—Speak English 


18,164 


233 


242 


267 


171 


1,416 


—Write in English 


18,100 


234 


241 


268 


170 


1,416 


—Use Math 


17,984 


234 


241 


269 


166 


1,416 


—Work as Part of aTeam 


18,102 


235 


242 


269 


172 


1,416 


—Solve Problems/Use Reasoning 


17,983 


235 


239 


266 


168 


1,416 
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TABLE B.3a 



NUMBER OF CASES USED IN CALCULATING DATA FOR TABLE B.3 
(PARTICIPANTS’ COURSE ENROLLMENT, ATTENDANCE, AND COMPLETION) 





National 
: Estimate 


PH 


||i|||l 


§pite3 : | 


Site 4 


|site ; ;||| 


Number of Courses Taken 


21,168 


269 


254 


278 


174 


1,636 


Completed at Least One Course 


21,168 


269 


254 


278 


174 


1,636 


Hours of Instruction Received 


9,769 


99 


175 


126 


62 


85 


Enrolled in Course with Various 
Instructional Methods Used 
“Frequently” or “Always” 
—Team Learning 


19,474 


269 


254 


270 


159 


1,454 


-Teacher-Led Classroom 


19,573 


269 


254 


270 


159 


1,454 


-Computer-Assisted Learning 


19,184 


269 


254 


270 


159 


1,454 


—Self-Paced Learning 


19,448 


269 


254 


270 


159 


1,454 


—Audio Visuals 


19,478 


269 


250 


270 


159 


1,454 


—Workplace Documents/Displays 


19,403 


269 


254 


270 


159 


1,341 


Enrolled in Course with Various 


Primary Emphases 


27,018 


340 


310 


401 


185 


1,455 


Enrolled in Course with Various 


Skills Taught 


20,838 


269 


254 


271 


159 


1,636 


Enrolled in Course that Met 


Various Times of Day/Week 


20,838 


269 


254 


271 


159 


1,636 



TABLE B.4a 



NUMBER OF CASES USED IN CALCULATING DATA FOR TABLE B.4 
(COURSE CHARACTERISTICS) 



National;! 

Estimate | Site | % . Site 2 $ite : : |p|^ite 4 - Site 5 



Instructional Methods Used 




-Team Learning 


1,851 


—Teacher-Led Classroom 


1,851 


—Computer-Assisted Learning 


1,851 


—Self-Paced Learning 


1,851 


-Audio Visuals 


1,851 


—Workplace Documents/Displays 


1,851 


Primary Emphasis 


1,843 


Skills Taught 


2,012 


Curriculum Uniquely Developed 


1,832 


Number of Learners Enrolled 


1,942 


Assessment Methods Used 


2,012 


Times of Day /Week Course Met 


2,022 



18 


17 


28 


17 


18 


18 


17 


28 


17 


18 


18 


17 


28 


17 


18 


18 


17 


28 


17 


18 


18 


16 


28 


17 


18 


18 


17 


28 


14 


17 


18 


17 


27 


17 


18 


18 


17 


29 


17 


20 


18 


17 


29 


14 


18 


18 


17 


31 


18 


20 


18 


17 


29 


17 


20 


18 


17 


29 


17 


20 
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TABLE B.5a 



NUMBER OF CASES USED IN CALCULATING DATA FOR TABLE B.5 
(STAFF DEVELOPMENT AND INVOLVEMENT BY EMPLOYERS AND UNIONS, 
REGARDLESS OF PARTNERSHIP STATUS) 





National 


iliiil 


/Site 2 i. 


; '::»Site : 3 


||it|ll 


! Site 1:1 


Staff Development Offered 


61 


l 


1 


1 


1 


1 


Employers/Unions Provided or 


Paid for Transportation 


335 


l 


1 


6 


6 


31 


Employers/Unions Provided or 


Paid for Child Care 


335 


l 


1 


6 


6 


31 


Employers/Unions Provided 
Release Time or Other 
Incentives/Recognition 


370 


l 


1 


6 


7 


31 
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TABLE B.6a 



NUMBER OF CASES USED IN CALCULATING DATA FOR TABLE B.6 
(INSTRUCTORS’ CHARACTERISTICS) 







;Sitel | 


11 !!! | 


§Sp;:3 


'Site Jig 


■ Site 111 


Percent Female 


57 


3 


4 


4 


4 


23 


Race/Ethnicity 


57 


3 


4 


4 


4 


23 


Prior Educational Attainment 


57 


3 


4 


5 


4 


23 


Prior Experience 


57 


3 


5 


5 


4 


24 


Percent with ESL Training 


57 


3 


5 


5 


4 


24 


Percent Bilingual 


57 


3 


5 


5 


4 


24 


Percent with State Teaching 
Certificate 


57 


3 


5 


5 


4 


24 
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APPENDIX C 



REVIEW OF EFFORTS TO DEVELOP APPLIED PERFORMANCE 
ASSESSMENTS WITH IN-DEPTH IMPACT SITES 



Introduction 



The appropriateness of using standardized basic skills assessments for measuring learners’ 
growth in workplace literacy programs is an issue that has been addressed in reviews of workplace 
evaluations (Kutner, Sherman, Webb & Fisher, 1991; Mikulecky & D’Adamo- Weinstein, 1991) and 
in discussions among practitioners. A number of reasons have been cited concerning the inadequacy 
of standardized tests. Workplace courses often are brief (sometimes two to five hours per week for 
six to eight weeks) and involve small groups of learners. Since standardized tests measure general 
literacy ability that usually improves only with a substantial intervention, they may not be sensitive 
to the measurement of changes over short periods of instruction. Furthermore, workplace literacy 
curricula are likely to address skills and competencies that are specific to a workplace rather than 
a broad range of general literacy skills. Thus, standardized tests of general literacy ability are not 
likely to show much change in learners’ performance in workplace-related literacy given the content 
of these tests. In light of these reasons, it has been suggested that custom-designed applied 
performance assessment measures should be considered in assessing what is taught in workplace 
programs (Mikulecky, 1994). 

While custom-designed measures have the potential of being more accurate assessments of the 
content and processes that are taught in workplace programs, there are issues to consider in their use. 
Workplace-specific measures, while providing important information on each workplace, are 
different from each other, thus making it difficult to conduct cross-site analyses of learner gains. 
The process of instrument development also requires a thorough understanding of the constructs 
being assessed so that comparable pre- and post-forms of the assessments can be developed. 
Another related issue concerns the focus of the assessment. When multiple skills are being taught 
in a workplace program, choices may need to be made concerning the key skills that should be 
assessed and the contexts in which they are measured. The process of selecting the skills and 
contexts for assessment and then constructing reliable and valid items requires knowledge about the 
processes for assessment design. 

During the evaluation of the National Workplace Literacy Program (NWLP), the issue of the 
limitations of standardized tests of general literacy ability was discussed by the evaluation staff and 
the members of the evaluation’s Advisory Committee. In light of these discussions and the U.S. 
Department of Education’s interest in applied performance assessment, a task was added to the 
evaluation to test the feasibility of having the In-Depth Study (IDS) sites develop and pilot test 
workplace-specific measures. The subcontractor for the evaluation (initially COSMOS Corporation 
and subsequently Abt Associates) had the task of providing technical assistance to the three sites in 
their applied performance assessment activities. 



Assessment Development 




Technical Assistance Activities. The NWLP evaluation subcontractor provided assistance to 
each of the three IDS sites in a number of areas. The sites were given examples of applied 
performance assessment instruments to use as a guide in developing their own instruments. The 
subcontractor also discussed the courses that might be the focus of the applied performance 
assessment and potential topics (i.e., competencies/skills) that could be assessed in these courses 
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using an applied assessment methodology. Since the sites differed in their knowledge about applied 
performance assessment methods and experience in creating job-specific literacy instruments, and 
were at various stages in developing a competency framework to guide the program's instructional 
content, the discussions between the subcontractor and the sites were held over many months as the 
sites made progress in organizing and refining the curriculum. 

As the sites prepared draft competency frameworks and instruments, the subcontractor reviewed 
these and provided feedback to the sites. This feedback concerned the appropriateness of the content 
of the instrument for assessing the specified competency, the format of the instrument, and the 
criteria for rating the assessments. The goal was for each site to develop and pilot test pre- and post- 
instruments, and then to refine the instruments based on the results of the pilot test. The sites all 
produced instruments, but performed the development and pilot-testing steps in varying degrees. 
Each site’s work on the applied performance assessment task is described below. 

Site 1. The workplace literacy staff at Site 1 had some previous experience with job-specific 
literacy through developing a written test for assessing learners’ knowledge of garment-related 
terms. As part of the applied performance assessment task, the subcontractor reviewed the garment- 
related pre-test and made a number of suggestions regarding its content and format. The staff at Site 
1 revised the pre-test and developed a post-test, which the subcontractor also reviewed. 

In addition to the garment-related test, the project director and instructor at Site 1 developed two 
other competency assessments. The first, Reading A Pay Stub, was designed to measure the skills 
of learners at the ESL and Basic Skills levels in reading and interpreting charts and tables. The 
second competency assessment was Reading A Spec Sheet, which was designed to measure the same 
skills for Basic Skills Level learners. Each of the assessments was pilot tested by the staff at Site 
1 . One hundred and thirty-one participants completed Reading a Pay Stub, which contained ten 
questions concerning information on a pay stub. Seventy-one participants completed the Reading 
A Spec Sheet assessment, which also contained ten questions. The participants’ responses on the 
assessments were reviewed to determine whether the responses were due to the wording of the 
question or the knowledge of the respondent. Revisions were made based on this review. A second 
form for each of the assessments also was developed. 

Site 2. The development of applied performance assessments at Site 2 was part of the 
curriculum development process that the instructional staff were involved with throughout the 
project. This process began with the identification of competencies for each of the core courses— 
Communication for the 90’s, Team Survival Skills, and Applied Math. Once the competencies were 
identified, the staff drafted and pilot tested the content for these courses and made several revisions 
in the content over the three years. Part of this process included piloting applied performance 
assessment items to measure group participation and problem-solving skills for the Team Survival 
Course. An assessment also was developed for the Applied Math course, in which the items were 
written to measure the application of math processes to the site’s workplace. Rather than conduct 
a formal pilot test, the staff administered the assessments and then revised them throughout the 
course. By the end of the project, the Team Survival Course included three modules: 1) Presentation 
Skills, 2) Problem Solving Skills, and 3) Group Participation Skills. Assessments were developed 
for each of these modules. 



The Presentation Skills module has only a post-assessment containing written and oral 
components. For the written component, class participants are asked to produce four pieces of 
writing related to the workplace, such as introductory remarks about a co-worker at a conference, 
an acceptance speech at an awards ceremony, an informative speech about safety, and a persuasive 
speech about the company’s benefits program. For the oral component, participants are asked to 
give a speech in each of three categories: impromptu, persuasive, and informative. 

The Problem Solving Module contains a pre- and post-test that assesses participants' ability to 
identify a problem, three possible solutions to the problem, the advantages and disadvantages of each 
of the possible solutions, and the solution that would best address the problem. Different scenarios 
are presented in the pre- and the post-tests. The Group Participation Skills module also contains a 
pre- and a post-test. Three different scenarios are presented on each of these tests that ask 
participants to write descriptions of what they would say in response to group situations. The 
assessments have been incorporated into the three modules that constitute the Team Survival course. 

Site 3. The workplace literacy project’s evaluator assumed responsibility for developing the 
applied performance assessments at Site 3. The assessment instruments were developed to: 1) 
determine the participant’s level of ESL for initial instruction, 2) indicate progress and mid-course 
placement, and 3) assess participants’ attitudes. For placement in initial instruction, participants 
were asked to complete a short job application form in addition to the standardized assessment tests 
that they were given. 

Applied performance assessments were created to determine participants’ progress in two levels 
of instruction: Level I: Initial ESL Instruction and Level II: Functional ESL Instruction. Five pre- 
and post-tests were developed for Level I in the following five competency areas: 1 ) Health and 
Safety, 2) Personal Health, 3) Finances, 4) Job Search and Promotion Skills, 5) Job Keeping. The 
pre-test items were the following: 

• Look at a picture and identify parts of the body; 

• Read instructions and indicate the time on different clocks; 

• Read a notice posted at work and answer questions about the notice; 

• Read a check stub and answer questions about it; and 

• Read a map of a plant and answer questions about the map. 
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Five items were developed as the post-test for Level I and as the pre-test for Level II. These were: 

• Read signs and write the meaning of each sign; 

• Look at the picture of body parts and complete sentences identifying the body parts; 

• Read a check stub and answer questions about it; 
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• Read a job ad and answer questions about the ad: and 

• Read a map of a plant and give oral directions for going from one area in the plant to 
another. 



A third set of assessment items was created as the post-test for Level II. These were: 

• List 3 tools or machines used in your area of work; list 2 articles of clothing that you are 
required to wear at your job; 

• Complete an accident report; 

• Write three sentences to explain an accident using terms listed in each of three columns; 

• Write an explanation of each sign shown below; and 

• Answer questions about the time you should do the following at work. 



While there were multiple items for the pre- and post-tests, the post-tests for each level 
presented a problem because they included more difficult items than the pre-test and required more 
writing. In addition, one item in the post-test for Level I required an oral response. This method of 
response was neither pre-tested in Level I or post-tested in Level II (since the post-test for Level I 
also served as the pre-test for Level II). Thus the items were not always equivalent for the two forms 
of the test. 

The evaluator conducted a pilot test of the items. The pre- and post-tests for Level I were 
administered to 47 participants in the treatment group and to 39 participants in the delayed group. 
An analysis of the data indicated similar gains for the groups ( the experimental group gained 6.8 
points based on a total of 1 8 points and the delayed group gained 7.4 points between the pre- and 
post-tests). 

An assessment also was developed to assess participants’ attitudes. This Return on Investment 
(ROI) checklist contained 1 1 items concerning work-related reasons for participating in class. 
Respondents were asked to check the items that were most important to them on this checklist. 

These assessments were incorporated into the final curriculum for the project. 



Conclusions and Lessons for the Future 
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Conclusions. The projects’ process of developing applied performance assessments was 
problematic due to a number of factors. The staffs’ lack of experience with assessment in general, 
and with applied performance assessment in particular, made it difficult for them to identify the 
skills that should be assessed, appropriate methods for assessing these skills, and the range of 
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acceptable responses. Furthermore, for two of the projects this was the first workplace literacy grant 
and the process of developing a curriculum took more time than they had expected, which is needed 
in order to develop assessments. These projects also did not begin with a competency framework, 
which made it difficult to organize a workplace curriculum. Once the projects agreed to participate 
in the applied performance assessment task, they focused on competency development and used this 
framework in determining the skills that they wanted to assess. While the projects were able to 
create applied performance assessment items, these items were limited in the skills that they assessed 
and would need further pilot testing and refinement before they would be useful to other workplace 
programs. 

Another factor that made this a challenging undertaking was the types of skills that needed to 
be assessed and the methods that should be used for assessing these skills. Applied performance 
assessment typically involves multiple modes of demonstration, including written and oral methods. 
Assessments involving oral methods usually are more expensive and time-consuming to administer 
than written assessments. At Site 2, for example, the Team Survival Skills course focused on the 
development of presentation and group participation skills. Optimally, these skills should be 
assessed using oral methods that entail individual observations. Not only was the staffs’ 
inexperience with assessment development an issue, but the types of skills that needed to be assessed 
were not routine and required knowledge and experience in assessment methods. 

Lessons for the Future. The experience of undertaking this task suggests that individual 
project development of applied performance assessments is not viable for the reasons discussed 
above. Since workplace literacy projects aim to teach skills in the context of the workplace, applied 
performance assessments are considered more desirable and accurate measures of the skills being 
taught. However, this is a complicated process if quality assessments that are reliable and valid are 
to be produced. 

While the industries in which there are workplace literacy projects are varied, the range of skills 
are similar and increasingly include problem-solving, presentation, and group participation or team 
work skills. These skills usually are not measured by most standardized literacy assessments. One 
solution is to have a professional with knowledge of applied performance assessment methods 
develop a generic set of assessments that measure oral and written communication, problem-solving, 
and teamwork skills. These assessments then could be customized to industries by including 
applications from the specific industries. 

Other factors should be considered in the development and use of applied performance 
assessment. In order to measure change in learners’ skills, at least two equivalent forms of applied 
assessment items should be developed and pilot tested. 

• Since the process of developing these assessments is lengthy and expensive, applied 
performance assessment should be used primarily when standardized multiple-choice 
tests are not adequate to measure the skill being taught. 

• Any assessment development activity also should include the creation of assessment 
administration procedures and processes for rating the assessments. 
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• The length of time between assessments will vary depending on the complexity of the 
skills being taught and the skill levels of the learners receiving instruction. For 
programs offering only short courses (for example, five hours), applied performance 
assessment is not an economically viable alternative. This methodology would be better 
used with longer courses teaching more complex skills. 

• Staff training is an important element in considering the use of applied performance 
assessment. Given the historical reluctance to apply assessment in adult education, it 
is likely that most staff in a workplace literacy program would require comprehensive 
training in assessment to implement a quality assessment component. 
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PROFILE OF IMPACT STUDY DESIGN AND 
IMPLEMENTATION AT THE THREE LOCAL SITES 
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Site 1 

• Baseline data collection. At the beginning of each course cycle NWLP staff interviewed 
the applicants and gave them the baseline data form, a standardized literacy assessment 
(the JOHN test), and a applied literacy assessment. MPR did not collect baseline data for 
supervisors. 

• Random Assignment Based on the workers' test scores and the interview . NWLP staff 
sorted workers into six groups: ESL 1-3 and Basic Skills 1-3; those with lower scores 
were placed in the ESL group and those with higher scores were identified for the Basic 
Skills courses. Once workers were grouped, MPR randomly selected workers from 
within groups for assignment to either the treatment or the control group. 

• Control Group. Workers in the control group were held back from taking NWLP courses 
for about 1 8 weeks (one course cycle). 

• Follow-Up Data Collection. At the end of each course, workers were given the follow-up 
data form, a standardized literacy assessment (workers selected for ESL courses were 
given the JOHN test and workers selected for Basic Skills courses were given the NYC 
test), and the applied literacy assessment. All members of the control group were given 
the JOHN test. We did not ask supervisors to complete the supervisor’s follow-up form 
nor did we collect data from employers. 

• Problems Encountered in Using the Study Design. None. 

Site 2 



• Baseline Data Collection. All workers in the plant were given the short-form of a 
standardized literacy assessment (CASAS locator test) and the baseline data form before 
classes started. Supervisors were given the supervisor's form to complete. 

• Random Assignment Based on workers’ scores on the CASAS locator test, NWLP 
staff sorted workers into groups for random assignment for two types of courses— 
reading and math. 1 Once sorted into the two groups, workers within shifts were selected 
for NWLP courses and the control group. 

• Control Group. Workers were assigned to the control group for 24-28 weeks. 




1 Some workers also were sorted into a group for critical thinking skills workshops/courses. In 
general, these workers were not given NWLP services during the period for which the grant 
operated. Site 2 plans to offer these workshops once the program has become institutionalized, after 
the grant is over. 




Follow-Up Data Collection. At the end of the courses workers were given the follow-up 
data forms and a standardized literacy assessment (CASAS ECS): only those workers 
who were in the math course were given the literacy assessment. Supervisors were given 
supervisor's forms to complete and MPR collected data on all study participants from the 
employer. 

Problems Encountered in Using the Study Design. It was originally planned that Site 
2 would select about 200 workers for the treatment group and about 200 workers for the 
delayed treatment group. They were unable to achieve this goal because many more 
members of the control group than planned came forward, after the delay period, to take 
NWLP courses. Since these workers were not subjected to random assignment a second 
time and they were given priority when making course assignments, the site had fewer 
openings for workers that could be randomly selected (that is, workers who were new 
applicants for NWLP services). The final number of workers selected for the treatment 
and control groups was 93 and 93, respectively. 



Baseline Data Collection. At the start of each course cycle workers were given the 
baseline data form and a standardized literacy assessment (initially, the LAS test was used 
and later the CASAS listening and reading tests were used). Supervisors were given the 
supervisor's form. 

Random Assignment Based on scores for the standardized literacy assessment, workers 
were sorted into groups (for example. Beginning ESL and Intermediate ESL). Groups 
were further refined to reflect where workers were employed because courses were held 
at the employer sites. Random selection for the treatment and the control group took 
place within test score/employer site groups. 

Control Group. Initially, workers were placed in the control group for 15 weeks. After 
the first two rounds of random selection, site staff asked if they could use a shorter delay 
period because of issues raised by employers and workers concerning the length of time 
before workers could participate in the NWLP courses. Given their concerns we reduced 
the delay period to one course cycle (8-9 weeks). 

Follow-Up Data Collection. Follow-up data were collected after two course cycles for 
workers who were selected in the first two rounds of random assignment. After the 
design was adjusted, to allow for a shorter delay period, we collected data from workers 
at the end of each course cycle. Workers were given the follow-up form and the 
standardized literacy assessment (during the first part of the evaluation the site used the 
LAS test and later switched to the CASAS). Supervisors completed the supervisor’s 
form. 

Problems Encountered in Using the Study Design. For many workers. Site 3 did not use 
a locator test before giving the baseline literacy assessment and may have used the wrong 



test level. This problem was not identified until several months after the evaluation 
started. Another problem encountered by Site 3 concerned maintaining the integrity of 
the treatment/control group design. We learned that instead of holding workers in the 
delayed treatment group back from NWLP courseis for 15 weeks as originally planned, 
they were offered NWLP services after one course cycle ended instead of after two cycles. 
Members of the treatment group and control group for the first two rounds of random 
assignment were excluded from all analyses to preserve the integrity of the random 
assignment design. 
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This appendix focuses on data issues relevant to our analyses, including the baseline 
characteristics of study participants, response rates on various data collection instruments, 
comparisons of respondents and nonrespondents, sample weights and nonresponse adjustments, and 
characteristics of study participants by hours of instruction received. 



A. BASELINE CHARACTERISTICS 

As described in Chapter II, the three impact-study sites each served a unique clientele. Table 
E.l shows selected baseline characteristics of the 942 workers in the three sites. 



B. RESPONSE RATES 

Response rates on each of the seven data collection instruments varied between sites and 
between treatment and control group members, as shown Table E.2. For the three baseline data 
collection instruments (worker and supervisor forms, and standardized literacy assessment), response 
rates were consistently very high— close to or equaling 1 00 percent. Response rates on the follow-up 
data collection instrument showed greater variation by type and across sites. The low response rates 
across all sites for the follow-up literacy assessment reflect the fact that only workers who both took 
the appropriate test and had within-range test scores were included in the numerator for the response 
rate as respondents. The response rates for treatments and controls were generally similar. For Sites 
1 and 2, there are no statistically significant differences in response rates for these two groups. For 
Site 3, the baseline supervisor form response rate for controls (98 percent) is significantly higher 
than for treatments (92 percent). 



C. CHARACTERISTICS OF RESPONDENTS AND NONRESPONDENTS 

Table E.3 presents baseline characteristics of respondents and nonrespondents for each of the 
five data collection instruments for which the nonresponse rate was four percent or more, and 
indicates whether the differences between the two groups were statistically significant. Overall, this 
table reveals that the respondents and nonrespondents to certain data collection instruments 
(particularly the follow-up worker and supervisor forms and the employer form) differed 
significantly on many characteristics, especially in terms of literacy skills and prior education. For 
example, compared with respondents, nonrespondents to the follow-up worker form were less likely 
to be bom in the U.S., more likely to rate their English abilities as “poor,” and had lower educational 
attainment. These background differences are also present when comparing workers for whom we 
received employer and/or supervisor forms to those for whom we have no employer ana/or 
supervisor forms. 





E-l 



TABLE E.l 

WORKERS* BASELINE CHARACTERISTICS, BY SITE 



Age (Years) 

Education 
Less Than 9 Years 
9 to 1 1 Years 
12 or More Years 
Race/Ethnicity 
Hispanic 
Black 

Asian/Pacific Islander 
White 
“Other 

Bom in the U.S. 

Female 

Self-Assessed Ability to Read English 
Poor 
Fair 
Good 
Excellent 

Self-Assessed Ability to Understand English 
Poor 
Fair 
Good 
Excellent 

Self-Assessed Ability to Speak English 
Poor 
Fair 
Good 
Excellent 

Self-Assessed Ability to Write English 
Poor 
Fair 
Good 
Excellent 

Self-Assessed Ability to Work as Part of a Team 
Poor 
Fair 
Good 
Excellent 

Self-Assessed Ability to Use Math 
Poor 
Fair 
Good 
Excellent 

Self-Assessed Ability to Solve Problems/Use Reasoning 
Poor 
Fair 
Good 
Excellent 

Any Health, Sight, Hearing Problem 
Health Problem or Disability 
Seeing Problem 
Hearing Problem 

Have Children Under Age 6 in Household 
Employed 

Months at Current Job (if employed) 

Assigned to Treatment Group 

Sample Si ze 



Site 1 


Site 2 


Site 3 




40 


40 


31 


•• 


I8°/« 


0% 


62% 


a) 


38% 


7% 


21% 


•• 


44% 


93% 


17% 


•• 


1.5% 


2% 


97% 


• • 


0% 


46% 


1% 


a) 


98% 


0% 


0% 


a) 


0% 


50% 


1% 


a) 


0.5% 


2% 


1% 




0.3% 


99% 


8%~ 




92% 


5% 


38% 


• • 


40% 


2% 


68% 


• • 


55% 


21% 


24% 


• • 


5% 


50% 


7% 


• • 


0% 


27% 


1% 


a) 


43% 


1% 


58% 


•• 


55% 


13% 


29% 


• • 


3% 


58% 


9% 


• • 


0% 


29% 


4% 


a) 


51% 


1% 


66% 


•• 


47% 


15% 


24% 


•• 


2% 


59% 


5% 


• • 


0% 


26% 


4% 


a) 


68% 


3% 


77% 


•• 


28% 


23% 


16% 


•• 


4% 


56% 


4% 


•• 


1% 


18% 


3% 


•• 


21% 


2% 


18% 


•• 


35% 


9% 


20% 


• • 


36% 


60% 


39% 


•• 


8% 


29% 


24% 


• • 


22% 


4% 


29% 


•• 


43% 


31% 


40% 




30% 


55% 


21% 


• • 


5% 


10% 


9% 




20% 


2% 


15% 


•• 


48% 


31% 


28% 


mm 


26% 


54% 


31% 


mm 


5% 


14% 


26% 


mm 


6% 


21% 


10% 


• • 


2% 


5% 


3% 




2% 


8% 


7% 


• • 


3% 


13% 


3% 


•• 


26% 


36% 


60% 


• • 


78% 


100% 


94% 


a) 


k 30 


147 


32 


• • 


54% 


51% 


50% 




409 


186 


347 





** compare distributions we used a Bonferoni test, 
for all planned comparisons within a category. 



The nominal level of significance was set at .05 



ERIC 



?2 T est statistical significance not computed because of zero percent or 100 percent response in at 
least one comparison. K 1,1 ai 



E-2 



172 



RESPONSE RATES BY SITE, OVERALL AND FOR TREATMENT AND CONTROL GROUPS 
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Another variable on which respondents and nonrespondents differed notably was job tenure. 
For example, among workers employed at baseline, those for whom we received a follow-up 
supervisor form had been employed an average of 80 months at baseline, compared with only 1 5 
months for workers without follow-up supervisor forms. This finding suggests that employed 
workers with less tenure at baseline left their jobs sooner, often before data were collected from 
them, supervisors, and employers. 

It is important to note, however, that while there are many differences between respondents and 
nonrespondents, response rates are similar for treatments and controls (see Section B). 



D. SAMPLE WEIGHTS AND NONRESPONSE ADJUSTMENTS 

To ensure that the results of our analyses pertained to the population of all workers in the study 
and not just to the sample of workers who responded, we created nonresponse-adjusted sample 
weights, which we used in all impact analyses. Weights were derived by (1) considering the 
probability of a worker being selected for the treatment or control group, and (2) taking into account 
nonresponse for the data collection instruments. 

Sites often stratified workers into groups based on their shift and literacy skills before they were 
randomly assigned to NWLP and the control group. The probability of being selected for NWLP 
was pj (the number of workers selected from stratum j divided by the total number of workers in the 
stratum) and the probability of being selected for the control group was 1-pj. The sample weights 
are defined as the inverse of the probability of being selected for NWLP and for the control group. 
The sample weights used for the treatment and control groups in our analyses sum to the number of 
applicants within strata. That is, both the treatment and control group have the same composition 
as the pool of workers who applied for NWLP courses. 

Nonresponse-adjusted weights were created by multiplying each worker’s probability of being 
selected for NWLP (or the control group) by the probability that the worker was a respondent, and 
then by computing the inverse of this product of the two probabilities (that is, the conditional 
probability of being selected and responding). We obtained the probability of responding from a 
logistic regression model that related a binary response variable to a large set of baseline 
characteristics. Some of the worker characteristics included in the model were self-reported literacy 
skills, standardized literacy test scores (if available), employment status, years of schooling in the 
U.S. and outside the U.S., sex, and treatment/control status. Statistical models were estimated 
separately by site and by data collection form, so that a total of four nonresponse-adjusted weights 
were created (follow-up worker form, follow-up supervisor form, employer form, and follow-up 
literacy assessment). We assigned workers who did not have a follow-up data collection form a 
weight of zero for the impact analyses. 



E. CHARACTERISTICS OF PARTICIPANTS BY HOURS OF INSTRUCTION 
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Chapter III described the effects of hours of instruction on learner outcomes. The analyses 
included only members of the treatment group who had one or more hours of instruction. Table E.4 
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TABLE E.4 



BASELINE CHARACTERISTICS OF TREATMENT GROUP. BY HOURS OF INSTRUCTION 



Age (Years) 

Education 
Less Than 9 Years 
9 to 1 1 Years 
12 or More Years 

Race/Ethnicity 

Hispanic 

Black 

Asian/Pacific Islander 

White 

Other 

Bom in the U.S. 

Site 
Site 1 
Site 2 
Site 3 

Female 

Self-Assessed Ability to Read English 
Poor 
Fair 
Good 
Excellent 

Self-Assessed Ability to Understand English 
Poor 
Fair 
Good 
Excellent 

Self-Assessed Ability to Speak English 
Poor 
Fair 
Good 
Excellent 

Self-Assessed Ability to Write English 
Poor 
Fair 
Good 
Excellent 



Comparison 1 Comparison 2 



Hours 

Valid 


Hours 

Missing 




Hours > 
Zero 


Hours = 
Zero 




37 


36 




38 




+ * 


32% 


31% 




26% 


48% 


** 


26% 


27% 




28% 


21% 




42% 


43% 




46% 


31% 


** 


34% 


50% 




19% 


75% 


** 


9% 


13% 




9% 


7% 




46% 


31% 




61% 


5% 


** 


10% 


6% 




10% 


11% 




1% 


0% 


a) 


1% 


2% 




20% 


28% 




19% 


21% 




47% 


34% 




62% 


5% 


** 


19% 


19% 




19% 


19% 




34% 


47% 




19% 


75% 


** 


58% 


45% 




67% 


35% 


** 


43% 


41% 




38% 


55% 


** 


39% 


28% 




44% 


24% 


** 


13% 


29% 




14% 


11% 




5% 


2% 




3% 


9% 




41% 


46% 




37% 


49% 




39% 


23% 


** 


45% 


25% 


** 


14% 


23% 




14% 


15% 




6% 


8% 




4% 


11% 


** 


47% 


49% 




46% 


52% 




34% 


17% 


** 


37% 


25% 


** 


13% 


28% 




13% 


12% 




6% 


7% 




4% 


10% 




60% 


55% 




60% 


58% 




23% 


17% 




24% 


23% 




12% 


26% 


** 


13% 


10% 




5% 


2% 




3% 


9% 


** 
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TABLE E.4 (continued) 





Comparison 1 


Comparison 2 






Hours 


Hours 


Hours > 


Hours = 






Valid 


Missing 


Zero 


Zero 




Self- Assessed Ability to Work as Part of a Team 










Poor 


17% 


19% 


15% 


22% 




Fair 


28% 


12% ** 


31% 


19% 


** 


Good 


41% 


53% 


42% 


39% 




Excellent 


14% 


16% 


12% 


20% 




Self-Assessed Ability to Use Math 












Poor 


21% 


17% 


19% 


27% 




Fair 


41% 


28% 


42% 


40% 




Good 


31% 


38% 


34% 


24% 


** 


Excellent 


6% 


17% 


5% 


10% 




Self-Assessed Ability to Solve Problems/Use Reasoning 










Poor 


15% 


9% 


15% 


17% 




Fair 


41% 


36% 


46% 


27% 


** 


Good 


33% 


31% 


33% 


33% 




Excellent 


11% 


24% 


7% 


23% 


♦ * 


Any Health, Sight, Hearing Problem 


11% 


9% 


10% 


12% 




Health Problem or Disability 


3% 


2% 


4% 


3% 




Seeing Problem 


4% 


7% 


3% 


9% 




Hearing Problem 


5% 


0% a) 


6% 


3% 




Have Children Under Age 6 in Household 


39% 


46% 


33% 


58% 


** 


Employed 


87% 


94% 


85% 


94% 


** 


Months at Current Job (if employed) 


53 


57 


55 


51 




Sample Size 


497 


50 


365 


132 





compare distributions we used a Bonferoni test. The nominal level of significance was set at 
.05 for all planned comparisons within a category. 



a) Did not test because of zero percent response. 
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shows baseline characteristics of all treatment group members, comparing (1) workers with valid 
data on hours to workers with missing hours and (2) workers with hours of instruction greater than 
zero to workers with zero hours of instruction. There are statistically significant differences between 
the two groups for both comparisons. For example, for four of the seven self-reported ability 
measures, workers with missing data for hours of instruction had generally higher self-reports than 
did those with valid hours data. In addition, compared to workers with positive hours of instruction, 
workers with zero hours of instruction had lower educational attainment and self-reported abilities, 
and were more likely to have children under age six in the household. The latter finding suggests 
that the presence of young children may affect a worker's ability to attend workplace literacy classes. 



APPENDIX F 

ESTIMATION OF PROGRAM IMPACTS AND THE EFFECTS OF 
HOURS OF INSTRUCTION ON WORKER OUTCOMES 





A. ESTIMATING PROGRAM IMPACTS 



Within each site, we randomly assigned workers to NWLP (the treatment group) and a control 
group. While the treatment and control groups, in theory, should be statistically equivalent, 
empirically they may differ with respect to one or more important background characteristics 
associated with worker outcomes. Finding such differences between the treatment and control 
groups for a few characteristics does not mean that the random assignment procedures were 
compromised; rather, they can occur by chance when randomly assigning workers to the treatment 
and control group. In. fact, we would expect to find that if we compared the characteristics of 
workers in the two groups that five percent of the comparisons would reveal statistically significant 
differences by chance alone. 1 

Table F.l shows the background characteristics of workers in the treatment and control groups, 
by site. Overall, there were very few differences between the treatment and control groups; however, 
there were a few noteworthy differences. Compared with the control group, Site 2 treatment group 
members had lower self-reported ability to solve problems. Compared with the control group, Site 
3 treatment group members had lower educational attainment, had lower self-reported abilities to 
work as a team and to solve problems, and wrote things at home less frequently for three of the 10 
items. Differences such as these may bias our estimates of program impacts. For Site 1, there were 
no differences between the two groups. 

To make the treatment and control groups as equivalent as possible and reduce possible biases 
in the impact estimates, we used a series of analytic models to compute program impacts. Besides 
adjusting for observed differences between the two groups, the estimates from the analytic model 
are generally more precise than those obtained by computing simple differences-of-means. The 
greater precision allows us to better assess whether impacts are statistically significant. Our analytic 
models included an indicator of treatment/control status and an estimate of the probability of being 
in the treatment group (a propensity score) as independent variables. By including the propensity 
score as an independent variable in the analytic model, we should obtain unbiased estimates of 
program impact under a reasonable set of assumptions. 2 



'This assumes that we have used a two-tailed test of statistical significance and have set the 
probability of rejecting the hypothesis of no difference by chance alone at .05. 

2 Besides using the analytic models to remove bias from our estimates of program impacts, we 
considered employing an approach that uses subclassification analysis where the subclasses are 
formed on the basis of the propensity scores and impacts are estimated within subclasses. The 
within subclasses impact estimates are then combined to form an estimate of program impact. 
Rosenbaum and Rubin (1984) show that this form of analysis is quite powerful and removes much 
of the bias that may be present even in a randomized experiment. We did not use this approach 
because of the relatively small samples for each site. Because we use an analytic model instead of 
the subclassification analysis, two key assumptions were necessary: (1) there was a linear 
relationship between the propensity score and workers’ outcomes and (2) the relationship between 
the propensity score and workers’ outcomes was similar for members of the treatment and control 

(continued...) 



We computed the probability of being in the treatment group (propensity score) for each worker 
in the sample by using a logit model that related a binary dependent variable (1 = in treatment group: 
0 = in control group) to a large set of independent variables. Some of these independent variables 
included age, sex, race/ethnicity, educational attainment, employment status, job tenure, score on 
the baseline literacy assessment (if available), and self-reported abilities. The variables used to 
predict the propensity scores varied across sites. For example, in Site 2 all of the people in the 
sample were employed when they filled out the baseline questionnaire, meaning we could not 
include employment status as a predictor of the probability of being selected for NWLP; in the two 
other sites some people were unemployed, allowing us to use employment status as a predictor. 
(Estimates for the propensity score models are shown in Tables F.2 through F.4). 3 

A more formal expression of the analytic model used to compute program impacts is: 

y =P +P T +P P +e 

where y is is the dependent variable (for example, follow-up standardized literacy score) for worker 
/' in site s , T i} shows whether a worker was in the treatment or control group (T - 1 if in the 
treatment group; 0 for the control group), is the estimated propensity (probability) of being in the 
treatment group, e u is the error term, and P 0l , P /j5 and P 2j are parameters to be estimated. The 
adjusted difference in means for the treatment and control groups (the program impact) corresponds 
to P /4 . This parameter shows the impact for workers in the treatment group and the control group 
who had the same probability of being selected for NWLP. We estimated the parameters of the 
analytic model with ordinary least squares when analyzing dependent variables that we measured 
on an interval or ordinal scale, and with maximum likelihood (a logit model) when computing 
program impacts for a binary dependent variable. Tests of statistical significance were based on one- 



2 (...continued) 

groups. We tested this last assumption using data for Site 1, the site with the largest sample of 
workers in the treatment and control group, and found in only seven of 60 models (all outcomes that 
were based on worker reports) used to estimate the impact of NWLP on worker outcomes, that a 
statistically significant interaction effect was present. Given the relatively few instances where a 
significant effect was observed, we chose to use a simpler model that did not include a variable 
allowing for different relationships between the propensity score and outcomes for workers in the 
treatment and control groups. 

3 To assess the adequacy of the logit model used to predict probabilities for each worker, we 
regressed each of the independent variables used in the logit equation on the estimated propensity 
score and the treatment/control status indicator. It was expected that if the propensity score does an 
adequate job of capturing the effect of the independent variables on the chances of being in the 
treatment group, then there should be no relationship between the independent variable and whether 
a worker was in the treatment/control group. This test of the propensity score model may show that 
the logit model was not well specified, for example, if we included only linear effects in the logit 
model when there should have been interactions between sets of independent variables or non-linear 
effects should have been estimated. The results of the specification test showed that in all cases the 
equation used to predict propensity scores was well specified. 

Wo 
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tailed t-tests that used estimates of program impact and standard errors of the estimates derived from 
the analytic model. All analytic models were estimated using the adjusted sample weights described 
in Appendix B. 



B. ESTIMATING THE EFFECT OF HOURS OF INSTRUCTION ON WORKER 
OUTCOMES 

To compute the effect of hours of instruction on workers’ outcomes, we examined the 
experiences of workers in the treatment group who had one or more hours of instruction. Because 
we do not have random assignment to help us sort out the true impact of instructional hours on 
worker outcomes, we used an analytic model to help us isolate the effects. Formally, we can write 
the analytic model as: 

H is =Z ,sY +e ,sH 



where H is corresponds to hours of instruction for worker / in project s ; Z is is a vector of background 
characteristics including the prior value of the outcome variable measured before the course started; 
prefers to an outcome variable measured at the time of the follow-up data collection; A^is a vector 
of background characteristics including the prior value of the outcome variable; e isH and e iJ} . are 
equation errors that are assumed to be correlated (that is, we allow for the presence of one or more 
unmeasured factors that may affect both hours of instruction and worker outcomes); and a, y , and 6 
are parameters (or vectors of parameters) to be estimated. The parameter of central importance to 
this phase of the analysis is a; this shows the change in the outcome for each additional hour of 
instruction. 4 

We estimate the outcome equation using the method of instrumental variables (Greene 1997). 
Instrumental variable estimates provide consistent estimates of the parameters in the outcomes 
equation under the assumption that, conditional on the other independent variables in the outcomes 
equation, one or more variables included in the hours’ equation are uncorrelated with the outcome 




4 Because we include the prior value of the outcome variable as one of the independent variables 
in the outcomes equation, we can interpret the parameters in the equation as showing their effect on 
changes in the outcome between the beginning and end of a course. This can be seen by using a 
simple example with just two independent variables: =a// /j +6y (j(; _ , ) +€ . , where the subscript 

t indicates a follow-up measurement for the outcome (/= 2) and the baseline measurement ( /— 1 = 1 ). 
By subtracting the baseline outcome score from both sides of the equation we obtain 
y is(2) -y^ , ) =&H is +(6-1 )y ll( , ) +€ . . This expression shows that acan be interpreted as the effect of 
hours of instruction on the change, for example, in a worker’s literacy skills between the time a 
course starts and the end of the course. 
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variable . 5 We included in the hours equation two variables that we excluded from the outcomes 
equation— number of children under six years old in the household and the date on which random 
assignment occurred. In almost all cases these variables (instruments) did not affect worker 
outcomes and affected hours of instruction. 



5 Parameter estimates are said to be consistent when they are unbiased and their variance is zero 
as the sample size becomes very large. 
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TABLE F.l 

BASELINE CHARACTERISTICS OF WORKERS IN TREATMENT AND CONTROL GROUPS. BY SITE 



Site 1 Site 2 Site 3 



Treatment 

Group 


Control 

Group 




Treatment 

Group 


Control 

Group 


Treatment 

Group 


Control 

Group 


Age (Years) 


40 


40 




40 


39 


31 


31 


Education 
















Less Than 9 Years 


17% 


19% 




0% 


0% a) 


68% 


56% 


9 to 1 1 Years 


41% 


34% 




5% 


8% 


18% 


23% 


12 or More Years 


42% 


47% 




95% 


92% 


14% 


21% 


Race/Ethnicity 
















Hispanic 


1% 


2% 




2% 


2% 


97% 


97% 


Black 


0% 


0% 


a) 


46% 


47% 


1% 


1% 


Asian/Pacific Islander 


98% 


98% 




0% 


0% a) 


0% 


0% 


White 


0% 


0% 


a) 


50% 


50% 


1% 


2% 


Other 


1% 


0% 


a) 


3% 


1% 


0% 


1% 


Bom in the U.S. 


1% 


0% 


a) 


99% 


99% 


10% 


6% 


Female 


91% 


93% 




6% 


4% 


40% 


36% 


Self-Assessed Ability to Read English 
















Poor 


39% 


41% 




2% 


2% 


70% 


66% 


Fair 


57% 


53% 




22% 


19% 


22% 


25% 


Good 


4% 


6% 




55% 


45% 


6% 


8% 


Excellent 


0% 


0% 


a) 


20% 


33% 


2% 


1% 


Self-Assessed Ability to Understand English 
















Poor 


42% 


44% 




1% 


0% a) 


62% 


55% 


Fair 


55% 


54% 




15% 


12% 


27% 


30% 


Good 


3% 


2% 




60% 


55% 


6% 


11% 


Excellent 


0% 


0% 


a) 


24% 


33% 


5% 


3% 


Self-Assessed Ability to Speak English 
















Poor 


52% 


49% 




1% 


0% a) 


66% 


66% 


Fair 


45% 


49% 




16% 


13% 


25% 


24% 


Good 


3% 


2% 




60% 


58% 


5% 


6% 


Excellent 


0% 


0% 


a) 


22% 


29% 


4% 


4% 


Self-Assessed Ability to Write English 
















Poor 


67% 


68% 




4% 


2% 


78% 


75% 


Fair 


28% 


29% 




26% 


20% 


15% 


18% 


Good 


4% 


3% 




53% 


59% 


5% 


4% 


Excellent 


1% 


1% 




17% 


19% 


3% 


3% 


Self-Assessed Ability to Work as Part of a Team 














Poor 


18% 


25% 




2% 


1% 


25% 


11% 


Fair 


37% 


33% 




10% 


9% 


21% 


19% 


Good 


36% 


35% 




63% 


58% 


39% 


39% 


Excellent 


9% 


7% 




25% 


32% 


16% 


32% 


Self-Assessed Ability to Use Math 
















Poor 


18% 


26% 




3% 


4% 


34% 


24% 


Fair 


42% 


45% 




36% 


26% 


39% 


42% 


Good 


34% 


26% 




53% 


58% 


18% 


24% 


Excellent 


6% 


4% 




8% 


11% 


9% 


10% 


Self-Assessed Ability to Solve Problems 
















Poor 


16% 


25% 




2% 


2% 


20% 


10% 


Fair 


53% 


44% 




39% 


22% ** 


25% 


31% 


Good 


27% 


26% 




45% 


63% ** 


33% 


29% 


Excellent 


4% 


6% 




14% 


13% 


22% 


30% 


How often write checks 
















Not at all 


14% 


12% 




10% 


12% 


62% 


54% 


Rarely 


20% 


18% 




10% 


16% 


13% 


15% 


Occasionally 


33% 


32% 




20% 


21% 


8% 


12% 


Regularly 


33% 


38% 




60% 


51% 


17% 


19% 
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TABLE F.I (continued) 



How often write notes or memos 
Not at all 
Rarely 
Occasionally 
Regularly 

How often write food recipes 
Not at all 
Rarely 
Occasionally 
Regularly 

How often write forms or applications 
Not at all 
Rarely 
Occasionally 
Regularly 

How often write appointments on a calendar 
Not at all 
Rarely 
Occasionally 
Regularly 

How often write letters 
Not at all 
Rarely 
Occasionally 
Regularly 

How often write stories or poems 
Not at all 
Rarely 
Occasionally 
Regularly 

How often write crossword puzzles 
Not at all 
Rarely 

Occasionally 

Regularly 

How often write grocery lists 
Not at all 
Rarely 
Occasionally 
Regularly 

How often write journal or diary 
Not at all 
Rarely 

Occasionally 

Regularly 

How often read letters or bills 
Not at all 
Rarely 

Occasionally 

Regularly 

How often read coupons 
Not at all 
Rarely 

Occasionally 

Regularly 



Site 1 SiteT Site 3 



Treatment 

Group 


Control 

Group 


Treatment 

Group 


Control 

Group 


Treatment 

Group 


Control 

Group 


36% 


35% 


8% 


9% 


47% 


43% 


28% 


29% 


21% 


21% 


25% 


30% 


23% 


20% 


38% 


30% 


17% 


16% 


14% 


15% 


33% 


41% 


11% 


11% 


47% 


52% 


62% . 


56% 


69% 


64% 


27% 


26% 


21% 


33% 


16% 


22% 


15% 


14% 


10% 


8% 


8% 


9% 


11% 


8% 


7% 


3% 


7% 


5% 


17% 


14% 


24% 


22% 


42% 


31% 


51% 


46% 


42% - 


43% 


24% 


28% 


25% 


31% 


22% 


23% 


23% 


34% 


7% 


9% 


12% 


12% 


10% 


7% 


22% 


18% 


16% 


20% 


41% 


29% 


26% 


28% 


25% 


29% 


27% 


29% 


28% 


26% 


33% 


34% 


18% 


27% 


25% 


29% 


26% 


17% 


13% 


14% 


23% 


23% 


32% 


43% 


26% 


15% 


33% 


31% 


45% 


36% 


26% 


34% 


22% 


24% 


21% 


17% 


31% 


30% 


22% 


22% 


2% 


4% 


17% 


20% 


61% 


63% 


67% 


63% 


75% 


71% 


28% 


25% 


23% 


21% 


13% 


17% 


9% 


9% 


7% 


12% 


6% 


6% 


2% 


3% 


3% 


3% 


6% 


6% 


57% 


59% 


59% 


51% 


72% 


65% 


33% 


30% 


20% 


34% 


15% 


24% 


9% 


9% 


18% 


13% 


8% 


7% 


1% 


2% 


4% 


2% 


4% 


4% 


36% 


34% 


37% 


31% 


46% 


39% 


25% 


29% 


22% 


28% 


17% 


25% 


17% 


18% 


23% 


31% 


17% 


14% 


22% 


19% 


17% 


10% 


20% 


21% 


52% 


53% 


85% 


81% 


73% 


60% 


22% 


25% 


9% 


9% 


10% 


20% 


17% 


15% 


1% 


6% 


7% 


8% 


9% 


7% 


5% 


4% 


10% 


12% 


12% 


14% 


3% 


1% 


15% 


16% 


24% 


26% 


3% 


6% 


18% 


13% 


28% 


23% 


25% 


14% 


17% 


23% 


36% 


37% 


69% 


79% 


50% 


49% 


19% 


22% 


24% 


19% 


53% 


47% 


29% 


26% 


26% 


19% 


23% 


21% 


21% 


22% 


25% 


33% 


13% 


17% 


31% 


30% 


25% 


28% 


12% 


15% 
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TABLE F. I (continued) 



Site 1 

Treatment 

Group 



How often read labels on food 

Not at all 26% 

Rarely 33% 

Occasionally 21% 

Regularly 20% 

How often read food recipes 

Not at all 37% 

Rarely 28% 

Occasionally 19% 

Regularly 16% 

How often read religious materials 
Not at all 56% 

Rarely 27% 

Occasionally 12% 

Regularly 5% 

How often read instructions 

Not at all 33% 

Rarely 35o/ 0 

Occasionally 21% 

Regularly 11% 

How often read street signs 

Not at all 16% 

Rarely 24% 

Occasionally 26% 

Regularly 34% 

How often read newspapers 

Not at all 190^ 

Rarely 16% 

Occasionally 1 50^ 

Regularly 50% 

How often read notes from teacher or school 
Not at all 22% 

Rarely 28% 

Occasionally 22% 

Regularly 29% 

How often read TV Guide or other television listing 
Not at all 25% 

Rarely 28% 

Occasionally 25% 

Regularly 23% 

How often read magazines 

Not at all 19% 

Rarely 22% 

Occasionally 27% 

Regularly 3 2 o/ 0 

How often read books 

Not at all 15% 

Rarely 21% 

Occasionally 27% 

Regularly 27% 

Any Health, Sight, Hearing Problem 5% 

Health Problem or Disability 2% 

Seeing Problem \o/ 0 

Hearing Problem 3% 

Kids Under 6 in Household 27% 
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Site 2 


Site 3 


Control 


Treatment 


Control 


Treatment 


Control 


Group 


Group 


Group 


Group 


Group 


27% 


13% 


8% 


42% 


37% 


29% 


16% 


17% 


20% 


24% 


24% 


26% 


30% 


16% 


16% 


20% 


45% 


45% 


22% 


23% 


40% 


32% 


33% 


61% 


59% 


27% 


36% 


25% 


18% 


22% 


19% 


17% 


28% 


10% 


7% 


14% 


15% 


14% 


11% 


12% 


58% 


14% 


10% 


45% 


36% 


24% 


27% 


28% 


23% 


28% 


14% 


34% 


28% 


15% 


15% 


4% 


26% 


34% 


17% 


20% 


31% 


4% 


3% 


27% 


21% 


34% 


17% 


19% 


22% 


25% 


21% 


29% 


34% 


24% 


25% 


15% 


51% 


44% 


27% 


29% 


14% 


5% 


4% 


34% 


32% 


31% 


6% 


4% 


7% 


11% 


24% 


11% 


8% 


12% 


10% 


31% 


78% 


83% 


47% 


47% 


18% 


4% 


0% a) 


33% 


31% 


18% 


7% 


7% 


29% 


24% 


18% 


15% 


15% 


19% 


21% 


47% 


74% 


78% 


19% 


23% 


25% 


35% 


23% 


48% 


40% 


26% 


27% 


16% 


19% 


22% 


22% 


17% 


31% 


17% 


18% 


27% 


22% 


29% 


16% 


20% 


26% 


15% 


11% 


54% 


46% 


26% 


16% 


16% 


20% 


21% 


28% 


14% 


10% 


13% 


14% 


20% 


54% 


62% 


13% 


19% 


17% 


4% 


3% 


37% 


29% 


23% 


15% 


8% 


27% 


28% 


26% 


29% 


30% 


22% 


22% 


34% 


52% 


59% 


15% 


21% 


17% 


12% 


12% 


30% 


30% 


23% 


26% 


24% 


28% 


26% 


24% 


34% 


36% 


23% 


21% 


36% 


28% 


29% 


19% 


22% 


7% 


22% 


20% 


12% 


9% 


2% 


6% 


5% 


3% 


3% 


3% 


8% 


8% 


9% 


5% 


3% 


14% 


11% 


2% 


3% 


24% 


36% 


35% 


59% 


62% 
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TABLE F.l (continued) 



Site 1 Site 2 Site3 





Treatment 

Group 


Control 

Group 


Treatment 

Group 


Control 

Group 


Treatment 

Group 


Control 

Group 


Employed 


80% 


77% 


100% 


100% a) 


93% 


96% 


Months at Current Job (if employed) 




32 


142 


152 


31 


33 


Reported Job Activities (if employed) 














Read Instructions 


37% 


43% 


84% 


84% 


43% 


44% 


Receive spoken instructions in English 


34% 


33% 


90% 


85% 


60% 


67% 


Speak English 


39% 


36% 


99% 


94% 


46% 


43% 


Work as pan of a team 


62% 


58% 


97% 


96% 


77% 


83% 


Write in English 


16% 


13% 


94% 


93% 


27% 


26% 


Use Math 


33% 


33% 


80% 


78% 


39% 


42% 


Solve problems/use reasoning 


46% 


44% 


93% 


92% 


65% 


65% 


Hours worked per week (if employed) 


39 


39 


41 


41 


40 


40 


Yearly earnings (if employed) 


SI 2,500 


SI 2,558 


S35.247 


S35.509 


SI 4.340 


SI 4.236 


Reported Benefits Received (if employed) 














Paid vacation 


40% 


37% 


98% 


100% a) 


67% 


63% 


Paid sick leave 


24% 


24% 


6% 


2% 


20% 


14% 


Paid holidays 


66% 


65% 


99% 


98% 


65% 


64% 


Health insurance 


52% 


51% 


97% 


96% 


65% 


63% 


Work at More than One Job (if employed) 


4% 


1% 


21% 


17% 


3% 


4% 


Sample Size 


260 


149 


93 


93 


194 


153 



**To compare distributions we used a Bonferoni test. The nominal level of significance was set at .05 for ail planned comparisons within 
a category. r 



a) Test of statistical significance not computed because of zero percent or 100 percent response. 
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TABLE F.2 



ESTIMATES FOR SITE 1 PROPENSITY SCORE MODEL 



Independent Variable 


Coefficient 


Z-Score 


Age less than 35 


-0.13 


-0.50 


Age greater than 45 


0.01 


0.05 


Female 


-0.09 


-0.21 


Less than 10 years of education 


-0.31 


-1.06 


More than 1 2 years of education 


-0.43 


-1.62 


Any health, sight, hearing problem 


-0.41 


-0.93 


Unemployed 


0.49 


1.12 


Hours worked per week greater than 40 


-0.00 


-0.01 


Hours worked per week less than 40 


-0.02 


-0.07 


Yearly earnings less than $8,000 


-0.04 


-0.15 


Yearly earnings greater than $15,000 


0.16 


0.53 


Months at current job less than 12 


-0.41 


-1.45 


Months at current job greater than 36 


-0.66 


-2.23 


Work at more than one job 


1.29 


1.62 


Score on baseline JOHN test 


-0.02 


-1.22 


General missing data indicator 


0.06 


0.09 


Index of self-assessed abilities 


0.09 


1.71 


Self-assessed ability missing data indicator 


0.30 


0.62 


Index of things written at home 


-0.00 


-0.01 


Writing index missing data indicator 


-0.40 


-0.82 


Index of things read at home 


-0.03 


-0.65 


Reading index missing data indicator 


-0.54 


-1.25 


Index of job activities 


-0.01 


-0.19 


Job activities index missing data indicator 


0.01 


0.02 


Constant 


2.02 


2.21 


Number of observations 




409 



NOTES: Dependent variable equals ! if worker was in the treatment group; otherwise it 
equals 0. Indices for writing, reading, and job activities variables created by summing 
workers’ values on each specific item. The corresponding missing data indicator equals 1 
if one or more response was missing. The general missing data indicator equals I if the 
worker has missing data for one or more of the remaining independent variables. 
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TABLE F.4 



ESTIMATES FOR SITE 3 PROPENSITY SCORE MODEL 



Independent Variable 


Coefficient 


Z-Score 


Age less than 25 


0.01 


0.04 


Age greater than 35 


0.07 


0.23 


Bom outside the U.S. 


-0.88 


-1.73 


Female 


0.24 


0.90 


Hispanic 


0.35 


0.54 


Less than 7 years of education 


-0.15 


-0.50 


Greater than 12 years of education 


-0.63 


-1.86 


Education predominantly in U.S. 


0.22 


0.50 


Any health, sight, hearing problem 


-0.05 


-0.12 


Unemployed 


1.55 


1.72 


Hours worked per week greater than 40 


-0.91 


-1.27 


Hours worked per week less than 40 


-0.19 


-0.38 


Yearly earnings (in thousands of dollars) 


0.07 


1.34 


Years at current job less than 1 


0.28 


1.01 


Years at current job 10 or more 


-0.74 


-1.21 


Work at more than one job 


-0.24 


-0.36 


Took CASAS test 


0.34 


0.99 


Randomization date 


0.00 


0.66 


General missing data indicator 


0.18 


0.29 


Index of self-assessed abilities 


-0.08 


-1.33 


Self-assessed ability missing data indicator 


-1.74 


-2.86 


Index of things written at home 


0.01 


0.18 


Writing index missing data indicator 


1.03 


1.64 


Index of things read at home 


-0.03 


-0.54 


Reading index missing data indicator 


-0.55 


-0.91 


Index of job activities 


0.02 


0.27 


Job activities index missing data indicator 


-0.18 


-0.30 


Constant 


-9.27 


-0.68 


Number of observations 




347 



NOTES: Dependent variable equals 1 if worker was in the treatment group; otherwise it 
equals 0. Indices for writing, reading, and job activities variables created by summing 
workers’ values on each specific item. The corresponding missing data indicator equals 1 
if one or more response was missing. The general missing data indicator equals 1 if the 
worker has missing data for one or more of the remaining independent variables. 
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